How big is the data because that changes how it's done. If we just added the fields and geometry to the datacards then had a function that you could pass the datacards JSON and it returned an interpolated field, for example, would that work? I think if it's not bigger than a few MB then JSON can work (where a few MB is a number coming from CouchDB and MongoDB people when they say after that storing binary data because can become slow... at very least not human readable).

The 3d field maps are big, the 2d field maps not so much.

Humm, thinking about it can I take the issue? I'm mucking about with the simulation infrastructure and this somehow sits in there at least until the detector stuff comes on...

I think my pseudo datacards addresses this, right?

Do they build field maps?

How big are the field maps? And do we store them as binary with some helper function in C++ with a swig interface for python?

4 bytes is a float. First question: if we run into space issues, we can ask the question of how many mantissa and exponent bits we need in the floating point representation.

Second question: can we just use a 2D map with a slowly varying error field? My naive studies of magnet measurements and wall stuff (the wall stuff repeated stuff I think you did) suggests this may be the case.

If we have to do 3D, then this:

4 bytes * 3 (15 m / cm) pi (1m / cm)^2

which samples every cm is equal to 1 GB, which means if we are smart we can do it on a single machine. But not parallelize to many cores.

Brainstorming...