CSV compatibility
interested
Pinky Nardo
The ability for VRChat to allow Unity to read CSV files for dynamic reading text into a world for better access. Junk files from crap like auto updating patreon lists cause it to break after a bit. Especially when reading from a JSON.
Log In
Phasedragon
Also, as a side note, what's breaking about JSON? Is there something wrong with the JSON parser provided to udon, or is it just that the script you are using to generate JSON has bugs and produces invalid JSON? If the latter, then I don't know how much it would help to switch to a different serialization scheme, if the root of the issue is that you have bad data.
Pinky Nardo
Phasedragon: The Issue that we were having is that the file produced junk files that were being feed into the data feed, our patreon list grabbed the names from the array it writes too so we can have our world give perks and update the board. The board suddenly stopped working after junk files reached its limit for JSON data meaning the board will no longer update. CSV however makes this more dynamic cause all it is, is calling back the names from a list instead of rewriting to a file and producing an output. If this makes any sense to you I don’t know.
Pinky Nardo
Is there a limit on VRC of how much json data can be recived from a string downloader (UdonSharp)?, and if so, how much. the board stopped working (as in only getting the first part of the data) when around 50,000 characters were received from what I expirenced.
Phasedragon
Pinky Nardo: the maximum size string you can download is 100mb, it doesn't sound like you were running into that. However the maximum size string you can sync over the network is right around 60kb. Accounting for multiple-bye characters, 50,000 characters is likely right there. But loading a Patreon list shouldn't require syncing over the network, as you can just let all the players download it themselves independently. If your system goes through the udonsync network then perhaps you should consider not doing that.
Honestly I don't see how CSV is going to help you with your problem. It would be useful to have, but it sounds like you need to do some cleaning up first, one way or another.
Pinky Nardo
Phasedragon: Our Syncing is local to the users, which is on a refresh of 1 hour, my partner makes the scripts for these, we don’t use the udon sync network as these scripts are manually created by us.
Pinky Nardo
Phasedragon: The JSON brings no errors to attention, since our scripts are made manually maybe it small mis format we have done somewhere, if we have we can look now to see if we can pinpoint and mitigate this issue, but I can confirm our file did reach over the 60KB threshold. We only mentioned the CSV being a viable option due to the fact it can help eliminate these junk overflows all together.
Phasedragon
Pinky Nardo: okay but what do you mean by junk overflows? That sounds like something you should just remove, and has absolutely nothing to do with whether it's JSON or CSV.
Pinky Nardo
Phasedragon: Some weird left over data coming from Patreon itself taking up storage for the limit of 60KB . If I think of it, might just be the way they work.
D
Docteh
Pinky Nardo: is patreon itself generating the json? Maybe you need a script to run some where to grab the data, parse it and host the processed file on GitHub.
Pinky Nardo
Docteh: The patreon is, we decided to go the alt approach and use Google sheet reference to dynamicly change it for every world at once with a more flexible script, I know this means adding the names to the sheet manually but it’s no big deal 🙂 it takes 30 seconds :D
Phasedragon
interested
This has been considered. Data lists/dictionaries were built with the intention to be flexible and support many different serialization standards. Right now we're working on byte array serialization, and then probably will do the inspector work. At some point, CSV would make a lot of sense too. The standard itself is quite simple and wouldn't be too hard to make a parser.
However, the big challenge with CSV is that the source does not describe what the end structure will look like. A bare bones implementation would be to just assume it's a list of lists: you've got X and Y coordinates and you're done. But what if you want to interpret the first entry in a row or column to be the key to a dictionary? Is this a list of dictionaries? A dictionary of lists? And there's also the matter of which direction you parse first. Should the rows be first or should the columns be first? There's a lot of unknowns.
Ultimately, with all these unknowns, it would make sense to simply allow the user to define it. But that would require sorting out all the different categories and figuring out a clean way for the user to define the structure, which is not trivial.
I'm curious what you think about this problem and how it should be approached.
Pinky Nardo
Phasedragon: Honestly I don’t know much myself. CSV I have minimal experience, only that it makes the most sense. CSV has so much flexibility that it could change at any moment meaning you have to re encode the parser every time a change is implemented. I can see why this can be a pain to the devs.
Phasedragon
Pinky Nardo: CSV doesn't change, it's a well defined standard. The problem is that it's a standard for spreadsheets, not data structures. As a result, it's ambiguous in how the spreadsheet should be interpreted as data structures, and requires the user to define that.