I'm amazed at developers who don't grasp that you don't need to have absolutely everything under the sun in a human readable file format. This is such a textbook case...
Yeah this isn't even human readable even when it's in YAML. What am I going to do? Read the floats and understand that the person looked left?
Even if you want it to be human readable, you don't need to include the name into every field and use balanced separators.
Any CSV variant would be an improvement already.
Even using C#'s decimal type (128bit) would be an improvement! I count 22 characters per numbers here. So a minimum of 176bit.
That's it everyone, back to copybooks.
Maybe use a real database for that? I'm a fan of simple tools (e.g. plaintext) for simple usecases but please use appropriate tools.
What is wrong with a file for this? Sounds more like a local log or debug output that a single thread in a single process would be creating. A file is fine for high volume append only data like this. The only big issue is the format of that data.
What benefit would a database bring here?
I think SQLite is a great middle ground. It saves the database as a single .db file, and can do everything an SQL database can do. Querying for data is a lot more flexible and a lot faster. The tools for manipulating the data in any way you want are very good and very robust.
However, I'm not sure how it would affect file size. It might be smaller because JSON/YAML wastes a lot of characters on redundant information (field names) and storing numbers as text, which the database would store as binary data in a defined structure. On the other hand, extra space is used to make common SQL operations happen much faster using fancy data structures. I don't know which effect is greater so file size could be bigger or smaller.
Smaller file size, lower data rate, less computational overhead, no conversion loss.
A 64 bit float requires 64 bits to store.
ASCII representation of a 64 bit float (in the example above) is 21 characters or 168 bits.
Also, if every record is the same then there is a huge overhead for storing the name of each value. Plus the extra spaces, commas and braces.
So, you are at least doubling the file size and data throughput. And there is precision loss when converting float-string-float. Plus the computational overhead of doing those conversions.Something like sqlite is lightweight, fast and will store the native data types.
It is widely supported, and allows for easy querying of the data.
Also makes it easy for 3rd party programs to interact with the data.If you are ever thinking of implementing some sort of data storage in files, consider sqlite first.
Because this is not log or debug data as OP said. In any case, what do you think would happen with this data? It will be analyzed by some sort of tool because no one could manually look at this much text data. In text, this can be like 1MB of data per second. So in a normal eye tracking session, probably hundreds of MB. The problem isn't the storage space, but the time it will take to read that in and analyze it each time, forcing you to wait for processing or use lots of memory while reading it. And anyway, in most languages, it's actually much easier to store the number values directly (in 8 bytes not the 30something this text representation uses) than to convert them to JSON, all languages have some built-in way to do that. And even if not, sqlite is piss-easy and does everything for you, being as simple as JSON.
There is just no reason to do it like that unless you just don't think about what you're doing or have no clue.
Some order in the CSV data, if it weren't a logfile, which i didn't know.
Fuck yaml. I'm not parsing data structured with spaces and newlines with my eyes. Use visible characters.
Does your viewer/editor not show space chars or indent levels?
Why you shouldn't use YAML
The best approach would be to never use yaml for anything
YAML doesn't require any level of accuracy for floating point numbers, and that doc appears to have numbers large enough to run into problems for single-precision floats (maybe double too). That means different parsers could give you different results.
i mean, json is valid yaml
I really like YAML but way too many people use it beyond its purpose... I work with Gitlabci and seeing complex bash scripts inline in YAML files makes me want to hurt people.
I’d probably just use line delimited JSON or CSV for this use case. It plays nicely with cat and other standard tools and basically all the yaml is doing is wrapping raw json and adding extra parse time/complexity.
In the end consider converting this to parquet for analysis, you probably won’t get much from compression or row-group clustering, but you will get benefits from the column store format when reading the data.
Thanks for the advice, but this is just the format of some eyetracking software I had to use not something I develop myself
Ah, well, such is software dependencies.
This is nasty to look at
Also let's represent all numbers in scientific notation, I'm sure that's going to make it easier to read...
