diff --git a/Reflection.md b/Reflection.md index 2e902ef..c6374f1 100644 --- a/Reflection.md +++ b/Reflection.md @@ -1,20 +1,20 @@ -# Issues +# Reflection -## Issues during testing +## Issues during testing (October 30th) ### Scheduler I don't know if it was the scheduler specifically, but whenever I try to test it, both the cache and the queue seem to become empty after the first function is completed. I don't know how to solve this yet -## Caching solution +## Caching solution (October 30th) ### Lack of data -Not being able to gather blocks of 50 is a huge issue. The idea of reusing blocks becomes less viable if I can just get a big chunk every time for others to use +Not being able to gather blocks of 50 is a big issue. The idea of reusing blocks becomes less viable if I can't just get a big chunk every time for others to use. I can't force a big chunk, because it might just return a status 1, and then the user gets nothing. Perhaps I can separate the questions myself and make a collection, that i can build the requests from. However, for every quest tracking if someone has already seen is a huge pain. -## Possible solution +## Possible solution (October 31st) Cache all data, 50 at a time Store it in a database When its done, let the db make blocks for quick cache access @@ -28,7 +28,7 @@ Then start replacing the cache items one at a time. Users shouldn't feel this at all, because the active sessions are stored in their own cache items -# Progress so far +# Progress so far (November 3rd) There only seem to be ~4100 items, making the collecting phase only last around 7 minutes. Swapping dbs is replaced with swapping tables, seems to work just as fine, yet much simpler to set up @@ -36,3 +36,28 @@ Swapping dbs is replaced with swapping tables, seems to work just as fine, yet m TODO: Add a caching state, and start filling up each unique combination with the items from the db This should lead to performance gains +## Possible future alternative +In the current state (when the cache is done), the backend might hold data 3 times (Cache, active DB table, and background DB table during the night) + +### Purely Cached solution +- Every night +- Check for each category+difficulty combination how many questions there are (API has an endpoint for this) +- Track which one you are on, and how many you have +- Take blocks of 50, until the remaining items are less than 50, then get that exact amount +- When they are gathered, either + 1. Divide them between 3 cache items: cat:dif:any/multiple/boolean + - (Manually filter on the type, since the API sends questions with either option together), or + 2. In one like cat:dif:any and then filter it by hand when a user asks for it + - (This will replace the collection put there the night before) +- Then move to the next one +- Stop when the last combination has been collected + +#### When a user requests data: +- Get the items from the cache +- Less items than requested: Inform the user to try a different combination +- More items than requested: Randomize and select a subset + +### Tradeoffs +- Advantage: Less total data held on the server + +- Downside: Slower gathering, since there are 25+3+4=300 unique combinations (each of which will end with a request of less than 50 items), it will require a decent number requests more than the bulk collecting that the DB does. \ No newline at end of file