2
$\begingroup$

Maybe this is obvious for some people but I'm a newbie when it comes to working with astronomical data. I have the same catalogue in different formats and am a bit surprised as to how sizes change. This is the breakdown:

  • catalogue.fits: 9.3 GB
  • catalogue.csv (saved as plain CSV): 14.9 GB
  • catalogue.csv (saved as ECSV): 18.3 GB

The original catalogue was ECSV (GaiaDR3). I made the conversion using stilts/topcat and loaded them into topcat to make sure they all have the same data.

I have googled this question but couldn't find any useful information, other than it is recommended to work with fits files when catalogues become too large.

Is this expected? I'm afraid I might have screwed something in the conversion even though the process was pretty straight forward.


EDIT: as per request in comments, I am pasting an extract of the CSV and ECSV files. These are the first 10 lines after the header:

CSV

1636148068921376768,Gaia DR3 34361129088,34361129088,894504938,2016.0,45.00432028915398,0.09731972,0.021047763781174733,0.101752974,3.235017271512856,0.12045025,26.857704,35.230515,29.518344127131527,0.13369285,19.231654938806578,0.13392176,0.16325329,6.428645E-4,-0.073663116,-0.012016551,-0.40389284,-0.10152152,-0.31593448,0.14065048,0.23142646,0.38175407,172,0,171,1,1.081467,194.59933,0.26741344,1.0328022,31,false,1.285487,,,,,,,,20,15,0.22053866,20,9,0,0.05737302,84.542816,0,0,1.0578898,0.28431648,0.18242157,0.4234895,0.8483561,-101.856606,-31.586445,-44.381237,29.909302,false,170,1763.191386728999,2.1212356,831.2096,17.571619,18,389.99713585371074,9.491409,41.089485,18.86089,19,2178.214858374066,15.074686,144.49487,16.402643,1.4565701,0,2,0,1,0,2.4582462,1.2892704,1.1689758,,,,,,,,,,,,,,,,,,,,,,,NOT_AVAILABLE,176.94278852482034,-48.88493355232444,42.54657309907107,-16.317212317623884,false,false,0,false,false,false,false,false,true,true,false,1.0820195E-13,5.682676E-13,0.9993886,3478.5408,3461.1475,3497.5784,4.7,4.6405,4.7734,-0.6143,-0.7064,-0.4964,302.2347,292.5325,312.6373,0.7643,0.7292,0.7975,0.505,0.4815,0.5273,0.3096,0.2956,0.3228,MARCS
1636148068921376768,Gaia DR3 38655544960,38655544960,1757259052,2016.0,45.004978371745516,0.017885398,0.019879675701858644,0.01877158,3.1391701154499523,0.022347411,140.47131,35.30821,29.686339048921702,0.023771733,19.115199913956804,0.023830384,0.1152631,0.07323115,-0.10691941,-0.03021361,-0.4488658,-0.15551351,-0.37927917,0.18184616,0.26367012,0.35528076,183,0,182,1,0.26434276,181.43846,0.0,0.0,31,false,1.4550159,,,,,,,,21,15,0.03929549,21,9,0,0.024301996,98.629005,0,0,1.012191,0.30656147,0.20578752,0.45299426,0.84656596,-96.31889,-34.497215,-44.82578,30.34742,false,180,42030.60043942405,11.392837,3689.213,14.128453,20,17955.47937733753,26.03932,689.55255,14.70305,19,34263.48754002838,36.75135,932.30554,13.410816,1.2424035,0,3,0,2,0,1.2922335,0.5745964,0.71763706,41.187176,3.1130338,2,10,1,8,7.034563,,,749.9199,,4500.0,3.0,-0.25,111,,,,13.068616,0.049816404,10,,NOT_AVAILABLE,176.94476211452783,-48.88527012426483,42.546872019115916,-16.318521975182243,false,false,0,true,true,false,false,false,true,true,false,1.03982646E-13,5.193881E-13,0.9998059,4708.7944,4659.062,4723.2773,4.5588,4.5261,4.5654,-0.087,-0.1218,-0.0681,332.8322,330.4709,347.1729,0.2345,0.184,0.2516,0.182,0.1425,0.1955,0.0961,0.0752,0.1032,MARCS
1636148068921376768,Gaia DR3 549755818112,549755818112,1176142027,2016.0,45.04828232129832,0.027803512,0.04825396034378256,0.026499804,1.5834770072004039,0.03442545,45.99728,16.465364,0.8431278207235642,0.03881713,-16.443764103221557,0.032919735,0.15041357,-0.14103404,0.058549184,0.17610951,-0.47409967,0.19906765,0.040113226,-0.14495842,-0.13733767,0.25881717,186,0,185,1,1.7361301,220.66844,0.072144866,0.96563005,31,true,1.4643211,,,,,,,,21,13,0.05741181,21,8,0,0.02840035,136.8429,0,0,1.0909767,0.3311718,0.18608999,0.49344972,0.84215754,-121.24405,-9.218482,-38.814762,29.860806,false,182,19047.581229390133,6.5483584,2908.7566,14.987767,18,8336.447382322891,16.801348,496.1773,15.53609,17,15362.299344786756,18.731024,820.15265,14.2817545,1.2441866,0,0,0,0,0,1.2543354,0.5483227,0.7060127,,,,,,,,,,,,,,,,,,,,,,,NOT_AVAILABLE,176.95936562964246,-48.83439750417775,42.59867306757699,-16.30407740663594,false,false,0,false,false,false,false,false,true,true,false,1.0493143E-13,5.2412805E-13,0.999712,4837.496,4819.394,4859.1646,4.4109,4.3971,4.426,-0.6022,-0.6293,-0.5715,621.2011,607.6643,634.6024,0.3668,0.3498,0.3862,0.2886,0.275,0.3042,0.155,0.1476,0.1633,MARCS
1636148068921376768,Gaia DR3 828929527040,828929527040,602360354,2016.0,45.02361979732255,0.054348446,0.06841876724959775,0.057792775,1.2030946627289945,0.066816084,18.006063,17.646046,13.952005440191227,0.078203134,-10.803908607898379,0.077209964,0.15176746,0.035847045,-0.17484911,-0.019222464,-0.43819016,-0.13019522,-0.38296714,0.18708444,0.24369827,0.3748652,180,0,174,6,0.67423594,189.07535,0.0,0.0,31,false,1.4280801,,,,,,,,21,15,0.12849106,21,9,0,0.021228304,100.85552,0,0,1.0349011,0.31510893,0.21111594,0.4635199,0.84135246,-94.50633,-34.891853,-45.02362,30.44096,false,175,4394.201551830577,3.3469398,1312.9012,16.580168,15,1758.2038783868772,16.53475,106.33386,17.225868,17,3789.788093274453,18.455858,205.34337,15.801358,1.2625711,0,2,0,0,0,1.42451,0.64570045,0.77880955,,,,,,,,,,,,,,,,,,,,,,,NOT_AVAILABLE,176.91130587619824,-48.838162803700186,42.58025775246733,-16.277574201332826,false,false,0,false,false,false,false,false,true,true,false,1.0353673E-13,5.1752347E-13,0.9998501,4333.865,4303.369,4382.055,4.6641,4.6466,4.6782,-0.3251,-0.3924,-0.2468,690.5604,669.148,719.1542,0.0405,0.0079,0.0907,0.0304,0.0059,0.0684,0.0162,0.0031,0.0363,PHOENIX
1636148068921376768,Gaia DR3 1275606125952,1275606125952,1616763991,2016.0,44.993270784169155,0.044207256,0.07633404499591856,0.037413534,0.6296499872212442,0.0480792,13.096099,6.749295,-1.4354337293932473,0.05779658,-6.594885755987001,0.046561327,0.017531538,0.15331538,-0.041159563,-0.02230982,-0.19973202,-0.025520978,-0.18153821,-0.0039606066,6.9630594E-5,0.075942636,204,0,204,0,0.22948465,211.85617,0.0679066,0.30813953,31,false,1.5075005,,,,,,,,24,15,0.08034339,24,10,0,0.019139638,149.57747,0,0,1.0098441,0.20883662,0.20302725,0.30122048,0.84553945,-73.91511,-2.6010761,-45.711555,29.540922,false,201,6031.684729758614,3.7787752,1596.2009,16.23627,21,2954.204025222018,15.563785,189.8127,16.662441,20,4378.056173763009,17.483109,250.41635,15.644692,1.2156239,0,2,0,2,0,1.0177488,0.42617035,0.5915785,,,,,,,,,,,,,,,,,,,,,,,NOT_AVAILABLE,176.87062873355376,-48.85450727416134,42.552445474053975,-16.26111514698619,false,false,0,true,false,false,false,false,true,true,false,1.0231336E-13,5.1105317E-13,0.99997115,5040.7686,5034.1396,5048.8696,4.4445,4.4181,4.4688,-0.6965,-0.7331,-0.6609,1343.5872,1283.9733,1404.766,0.004,9.0E-4,0.01,0.0032,7.0E-4,0.0081,0.0017,4.0E-4,0.0043,PHOENIX
1636148068921376768,Gaia DR3 1374389600384,1374389600384,866663434,2016.0,44.932802106149424,0.13405186,0.06480894307270345,0.116464846,1.7650550406017367,0.14989038,11.775639,11.373369,5.092345091384519,0.17400531,-10.169638649405274,0.14390641,0.012303141,0.1726108,-0.06496814,0.059556037,-0.20390691,0.035888158,-0.14015855,0.07941218,0.050555214,0.07260491,237,0,236,1,-1.963938,199.77853,0.0,0.0,31,true,1.2888188,,,,,,,,27,16,0.24213506,27,11,0,0.046150215,65.5374,0,0,0.9086372,0.21830778,0.20200518,0.28883445,0.8636228,-77.12932,3.91698,-47.065266,29.11508,false,233,849.1830497115145,1.5339631,553.58765,18.364864,24,174.8028102530512,8.37604,20.869385,19.732172,26,962.1708521083773,8.57319,112.23021,17.289764,1.338903,0,0,0,1,0,2.4424076,1.3673077,1.0751,,,,,,,,,,,,,,,,,,,,,,,NOT_AVAILABLE,176.81934898907144,-48.906005037340215,42.48870623058961,-16.25440588954115,false,false,0,false,false,false,false,false,true,false,false,1.9225565E-13,9.105778E-13,0.9999547,3404.9011,3392.6704,3433.1738,4.4294,4.3898,4.4884,-1.1774,-1.1924,-1.1333,426.0444,406.6133,445.8924,0.4993,0.4523,0.5627,0.3329,0.3016,0.3757,0.1986,0.1802,0.2236,MARCS
1636148068921376768,Gaia DR3 1619203481984,1619203481984,955831289,2016.0,44.95115803041135,0.11489181,0.10531247613400328,0.092665374,1.7172992921993269,0.11708278,14.667395,18.377512,13.217855646757254,0.15483287,12.767977485493745,0.11529025,-0.10194003,0.12885134,-0.26817575,0.17114455,-0.19460984,0.16064924,-0.19148935,0.05554392,0.046069436,-0.09006234,229,0,224,5,-0.09883537,226.36523,0.2717901,0.8230513,31,false,1.2836995,,,,,,,,26,16,0.21913311,26,11,0,0.014332649,110.389854,0,0,0.9937619,0.19364144,0.2579131,0.27664497,0.8563101,-81.96813,9.220485,-45.121784,29.153976,false,219,1329.8584660892113,1.9149755,694.45197,17.877853,24,280.1407226446208,8.440962,33.188248,19.220102,21,1573.835843510322,13.714526,114.75685,16.755497,1.3941157,0,3,0,0,0,2.4646053,1.3422489,1.1223564,,,,,,,,,,,,,,,,,,,,,,,NOT_AVAILABLE,176.79447406087172,-48.8646537946501,42.519355519308434,-16.221066034883595,false,false,0,false,false,false,false,false,true,true,false,1.06596135E-13,5.396279E-13,0.999561,3595.7598,3589.6238,3605.467,4.3742,4.3274,4.3944,-0.4565,-0.5201,-0.3926,696.264,663.922,721.3262,0.9776,0.9422,1.0066,0.652,0.6285,0.6718,0.394,0.3805,0.4049,MARCS
1636148068921376768,Gaia DR3 1717987078400,1717987078400,71332990,2016.0,44.98309734471892,0.16068149,0.09640645832988629,0.14490694,2.7367159160633827,0.17841817,15.338774,14.770221,2.7031271619485757,0.21807563,-14.520761539059677,0.18491341,0.07472117,0.1078753,-0.14565209,0.053458534,-0.1670114,0.046779558,-0.19464372,0.067384295,0.035748687,0.15527137,211,0,210,1,-1.6899631,225.55008,0.20050569,0.20059653,95,false,,1.1166438,0.046840165,-0.06460931,-0.051463306,-0.0800577,-0.12249996,-0.01685062,24,16,0.31002954,24,10,0,0.017960072,159.78165,0,0,0.9159248,0.23936273,0.15536572,0.27342117,0.85889024,-65.00193,-16.740282,-48.247776,29.825163,false,207,665.7283829462514,1.6562455,401.95032,18.629124,19,125.10427068321398,11.716241,10.677851,20.095362,20,871.0756987417084,12.082568,72.09359,17.397757,1.496376,0,2,0,0,0,2.6976051,1.466238,1.2313671,,,,,,,,,,,,,,,,,,,,,,,NOT_AVAILABLE,176.83794624758525,-48.84788093533632,42.548440578810926,-16.23894270538447,false,false,0,false,false,false,false,false,true,false,false,1.4650697E-13,5.7786913E-13,0.9998262,3173.697,3167.9666,3183.0564,4.9626,4.9521,4.9725,-0.0169,-0.0595,0.0102,313.3219,302.2305,321.4363,0.0844,0.0351,0.1558,0.0525,0.0218,0.0971,0.0365,0.0152,0.0672,MARCS
1636148068921376768,Gaia DR3 2512556873600,2512556873600,831324186,2016.0,45.07492471994457,33.54464,0.1108820420939314,24.830378,,,,,,,,,-0.1309745,,,,,,,,,,47,0,47,0,35.859825,1964.6633,119.27316,184.45139,3,false,,,,,,,,,7,6,120.0138,8,8,0,0.41287413,54.55582,0,0,,0.4272169,0.21570276,0.5622911,0.8563249,-75.8122,-1.6494459,-50.740902,30.197552,false,47,45.0438638293822,2.318626,19.426964,21.553278,5,296.537445766598,54.37354,5.4537086,19.158344,5,444.69570491458535,83.88305,5.3013773,18.127739,16.455807,0,0,0,0,0,1.0306053,-2.3949337,3.425539,,,,,,,,,,,,,,,,,,,,,,,NOT_AVAILABLE,176.91896393240253,-48.771755887594544,42.64430938681278,-16.251990466593394,false,false,0,false,false,false,false,false,false,false,false,0.004575429,0.036596417,0.9587768,,,,,,,,,,,,,,,,,,,,,,null
1636148068921376768,Gaia DR3 2821794351744,2821794351744,1389218613,2016.0,45.124152453120615,1.6364167,0.13677852527701542,1.5161866,,,,,,,,,-0.057091914,,,,,,,,,,162,0,145,17,85.27109,11890.783,11.945554,971.3325,3,false,,,,,,,,,18,12,3.0579515,20,5,0,0.17080484,85.50274,58,0,,0.3759727,0.1906163,0.49689272,0.8553549,-98.59949,-1.5183473,-46.745052,28.801044,false,162,1180.830086047288,13.438973,87.86609,18.006899,18,448.9534667774457,10.775265,41.66519,18.708038,16,2295.837741126928,23.04338,99.63112,16.345543,2.324459,0,2,0,1,0,2.3624954,0.70113945,1.661356,,,,,,,,,,,,,,,,,,,,,,,NOT_AVAILABLE,176.94249514790158,-48.718340373548315,42.70123151869663,-16.2416233995086,false,false,0,false,false,false,false,false,false,false,false,0.0,0.0,0.9105993,,,,,,,,,,,,,,,,,,,,,,null

ECSV

1636148068921376768,"Gaia DR3 34361129088",34361129088,894504938,2016.0,45.00432028915398,0.09731972,0.021047763781174733,0.101752974,3.235017271512856,0.12045025,26.857704,35.230515,29.518344127131527,0.13369285,19.231654938806578,0.13392176,0.16325329,6.428645E-4,-0.073663116,-0.012016551,-0.40389284,-0.10152152,-0.31593448,0.14065048,0.23142646,0.38175407,172,0,171,1,1.081467,194.59933,0.26741344,1.0328022,31,"False",1.285487,null,null,null,null,null,null,null,20,15,0.22053866,20,9,0,0.05737302,84.542816,0,0,1.0578898,0.28431648,0.18242157,0.4234895,0.8483561,-101.856606,-31.586445,-44.381237,29.909302,"False",170,1763.191386728999,2.1212356,831.2096,17.571619,18,389.99713585371074,9.491409,41.089485,18.86089,19,2178.214858374066,15.074686,144.49487,16.402643,1.4565701,0,2,0,1,0,2.4582462,1.2892704,1.1689758,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"NOT_AVAILABLE",176.94278852482034,-48.88493355232444,42.54657309907107,-16.317212317623884,"False","False",0,"False","False","False","False","False","True","True","False",1.0820195E-13,5.682676E-13,0.9993886,3478.5408,3461.1475,3497.5784,4.7,4.6405,4.7734,-0.6143,-0.7064,-0.4964,302.2347,292.5325,312.6373,0.7643,0.7292,0.7975,0.505,0.4815,0.5273,0.3096,0.2956,0.3228,"MARCS"
1636148068921376768,"Gaia DR3 38655544960",38655544960,1757259052,2016.0,45.004978371745516,0.017885398,0.019879675701858644,0.01877158,3.1391701154499523,0.022347411,140.47131,35.30821,29.686339048921702,0.023771733,19.115199913956804,0.023830384,0.1152631,0.07323115,-0.10691941,-0.03021361,-0.4488658,-0.15551351,-0.37927917,0.18184616,0.26367012,0.35528076,183,0,182,1,0.26434276,181.43846,0.0,0.0,31,"False",1.4550159,null,null,null,null,null,null,null,21,15,0.03929549,21,9,0,0.024301996,98.629005,0,0,1.012191,0.30656147,0.20578752,0.45299426,0.84656596,-96.31889,-34.497215,-44.82578,30.34742,"False",180,42030.60043942405,11.392837,3689.213,14.128453,20,17955.47937733753,26.03932,689.55255,14.70305,19,34263.48754002838,36.75135,932.30554,13.410816,1.2424035,0,3,0,2,0,1.2922335,0.5745964,0.71763706,41.187176,3.1130338,2,10,1,8,7.034563,null,null,749.9199,null,4500.0,3.0,-0.25,111,null,null,null,13.068616,0.049816404,10,null,"NOT_AVAILABLE",176.94476211452783,-48.88527012426483,42.546872019115916,-16.318521975182243,"False","False",0,"True","True","False","False","False","True","True","False",1.03982646E-13,5.193881E-13,0.9998059,4708.7944,4659.062,4723.2773,4.5588,4.5261,4.5654,-0.087,-0.1218,-0.0681,332.8322,330.4709,347.1729,0.2345,0.184,0.2516,0.182,0.1425,0.1955,0.0961,0.0752,0.1032,"MARCS"
1636148068921376768,"Gaia DR3 549755818112",549755818112,1176142027,2016.0,45.04828232129832,0.027803512,0.04825396034378256,0.026499804,1.5834770072004039,0.03442545,45.99728,16.465364,0.8431278207235642,0.03881713,-16.443764103221557,0.032919735,0.15041357,-0.14103404,0.058549184,0.17610951,-0.47409967,0.19906765,0.040113226,-0.14495842,-0.13733767,0.25881717,186,0,185,1,1.7361301,220.66844,0.072144866,0.96563005,31,"True",1.4643211,null,null,null,null,null,null,null,21,13,0.05741181,21,8,0,0.02840035,136.8429,0,0,1.0909767,0.3311718,0.18608999,0.49344972,0.84215754,-121.24405,-9.218482,-38.814762,29.860806,"False",182,19047.581229390133,6.5483584,2908.7566,14.987767,18,8336.447382322891,16.801348,496.1773,15.53609,17,15362.299344786756,18.731024,820.15265,14.2817545,1.2441866,0,0,0,0,0,1.2543354,0.5483227,0.7060127,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"NOT_AVAILABLE",176.95936562964246,-48.83439750417775,42.59867306757699,-16.30407740663594,"False","False",0,"False","False","False","False","False","True","True","False",1.0493143E-13,5.2412805E-13,0.999712,4837.496,4819.394,4859.1646,4.4109,4.3971,4.426,-0.6022,-0.6293,-0.5715,621.2011,607.6643,634.6024,0.3668,0.3498,0.3862,0.2886,0.275,0.3042,0.155,0.1476,0.1633,"MARCS"
1636148068921376768,"Gaia DR3 828929527040",828929527040,602360354,2016.0,45.02361979732255,0.054348446,0.06841876724959775,0.057792775,1.2030946627289945,0.066816084,18.006063,17.646046,13.952005440191227,0.078203134,-10.803908607898379,0.077209964,0.15176746,0.035847045,-0.17484911,-0.019222464,-0.43819016,-0.13019522,-0.38296714,0.18708444,0.24369827,0.3748652,180,0,174,6,0.67423594,189.07535,0.0,0.0,31,"False",1.4280801,null,null,null,null,null,null,null,21,15,0.12849106,21,9,0,0.021228304,100.85552,0,0,1.0349011,0.31510893,0.21111594,0.4635199,0.84135246,-94.50633,-34.891853,-45.02362,30.44096,"False",175,4394.201551830577,3.3469398,1312.9012,16.580168,15,1758.2038783868772,16.53475,106.33386,17.225868,17,3789.788093274453,18.455858,205.34337,15.801358,1.2625711,0,2,0,0,0,1.42451,0.64570045,0.77880955,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"NOT_AVAILABLE",176.91130587619824,-48.838162803700186,42.58025775246733,-16.277574201332826,"False","False",0,"False","False","False","False","False","True","True","False",1.0353673E-13,5.1752347E-13,0.9998501,4333.865,4303.369,4382.055,4.6641,4.6466,4.6782,-0.3251,-0.3924,-0.2468,690.5604,669.148,719.1542,0.0405,0.0079,0.0907,0.0304,0.0059,0.0684,0.0162,0.0031,0.0363,"PHOENIX"
1636148068921376768,"Gaia DR3 1275606125952",1275606125952,1616763991,2016.0,44.993270784169155,0.044207256,0.07633404499591856,0.037413534,0.6296499872212442,0.0480792,13.096099,6.749295,-1.4354337293932473,0.05779658,-6.594885755987001,0.046561327,0.017531538,0.15331538,-0.041159563,-0.02230982,-0.19973202,-0.025520978,-0.18153821,-0.0039606066,6.9630594E-5,0.075942636,204,0,204,0,0.22948465,211.85617,0.0679066,0.30813953,31,"False",1.5075005,null,null,null,null,null,null,null,24,15,0.08034339,24,10,0,0.019139638,149.57747,0,0,1.0098441,0.20883662,0.20302725,0.30122048,0.84553945,-73.91511,-2.6010761,-45.711555,29.540922,"False",201,6031.684729758614,3.7787752,1596.2009,16.23627,21,2954.204025222018,15.563785,189.8127,16.662441,20,4378.056173763009,17.483109,250.41635,15.644692,1.2156239,0,2,0,2,0,1.0177488,0.42617035,0.5915785,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"NOT_AVAILABLE",176.87062873355376,-48.85450727416134,42.552445474053975,-16.26111514698619,"False","False",0,"True","False","False","False","False","True","True","False",1.0231336E-13,5.1105317E-13,0.99997115,5040.7686,5034.1396,5048.8696,4.4445,4.4181,4.4688,-0.6965,-0.7331,-0.6609,1343.5872,1283.9733,1404.766,0.004,9.0E-4,0.01,0.0032,7.0E-4,0.0081,0.0017,4.0E-4,0.0043,"PHOENIX"
1636148068921376768,"Gaia DR3 1374389600384",1374389600384,866663434,2016.0,44.932802106149424,0.13405186,0.06480894307270345,0.116464846,1.7650550406017367,0.14989038,11.775639,11.373369,5.092345091384519,0.17400531,-10.169638649405274,0.14390641,0.012303141,0.1726108,-0.06496814,0.059556037,-0.20390691,0.035888158,-0.14015855,0.07941218,0.050555214,0.07260491,237,0,236,1,-1.963938,199.77853,0.0,0.0,31,"True",1.2888188,null,null,null,null,null,null,null,27,16,0.24213506,27,11,0,0.046150215,65.5374,0,0,0.9086372,0.21830778,0.20200518,0.28883445,0.8636228,-77.12932,3.91698,-47.065266,29.11508,"False",233,849.1830497115145,1.5339631,553.58765,18.364864,24,174.8028102530512,8.37604,20.869385,19.732172,26,962.1708521083773,8.57319,112.23021,17.289764,1.338903,0,0,0,1,0,2.4424076,1.3673077,1.0751,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"NOT_AVAILABLE",176.81934898907144,-48.906005037340215,42.48870623058961,-16.25440588954115,"False","False",0,"False","False","False","False","False","True","False","False",1.9225565E-13,9.105778E-13,0.9999547,3404.9011,3392.6704,3433.1738,4.4294,4.3898,4.4884,-1.1774,-1.1924,-1.1333,426.0444,406.6133,445.8924,0.4993,0.4523,0.5627,0.3329,0.3016,0.3757,0.1986,0.1802,0.2236,"MARCS"
1636148068921376768,"Gaia DR3 1619203481984",1619203481984,955831289,2016.0,44.95115803041135,0.11489181,0.10531247613400328,0.092665374,1.7172992921993269,0.11708278,14.667395,18.377512,13.217855646757254,0.15483287,12.767977485493745,0.11529025,-0.10194003,0.12885134,-0.26817575,0.17114455,-0.19460984,0.16064924,-0.19148935,0.05554392,0.046069436,-0.09006234,229,0,224,5,-0.09883537,226.36523,0.2717901,0.8230513,31,"False",1.2836995,null,null,null,null,null,null,null,26,16,0.21913311,26,11,0,0.014332649,110.389854,0,0,0.9937619,0.19364144,0.2579131,0.27664497,0.8563101,-81.96813,9.220485,-45.121784,29.153976,"False",219,1329.8584660892113,1.9149755,694.45197,17.877853,24,280.1407226446208,8.440962,33.188248,19.220102,21,1573.835843510322,13.714526,114.75685,16.755497,1.3941157,0,3,0,0,0,2.4646053,1.3422489,1.1223564,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"NOT_AVAILABLE",176.79447406087172,-48.8646537946501,42.519355519308434,-16.221066034883595,"False","False",0,"False","False","False","False","False","True","True","False",1.06596135E-13,5.396279E-13,0.999561,3595.7598,3589.6238,3605.467,4.3742,4.3274,4.3944,-0.4565,-0.5201,-0.3926,696.264,663.922,721.3262,0.9776,0.9422,1.0066,0.652,0.6285,0.6718,0.394,0.3805,0.4049,"MARCS"
1636148068921376768,"Gaia DR3 1717987078400",1717987078400,71332990,2016.0,44.98309734471892,0.16068149,0.09640645832988629,0.14490694,2.7367159160633827,0.17841817,15.338774,14.770221,2.7031271619485757,0.21807563,-14.520761539059677,0.18491341,0.07472117,0.1078753,-0.14565209,0.053458534,-0.1670114,0.046779558,-0.19464372,0.067384295,0.035748687,0.15527137,211,0,210,1,-1.6899631,225.55008,0.20050569,0.20059653,95,"False",null,1.1166438,0.046840165,-0.06460931,-0.051463306,-0.0800577,-0.12249996,-0.01685062,24,16,0.31002954,24,10,0,0.017960072,159.78165,0,0,0.9159248,0.23936273,0.15536572,0.27342117,0.85889024,-65.00193,-16.740282,-48.247776,29.825163,"False",207,665.7283829462514,1.6562455,401.95032,18.629124,19,125.10427068321398,11.716241,10.677851,20.095362,20,871.0756987417084,12.082568,72.09359,17.397757,1.496376,0,2,0,0,0,2.6976051,1.466238,1.2313671,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"NOT_AVAILABLE",176.83794624758525,-48.84788093533632,42.548440578810926,-16.23894270538447,"False","False",0,"False","False","False","False","False","True","False","False",1.4650697E-13,5.7786913E-13,0.9998262,3173.697,3167.9666,3183.0564,4.9626,4.9521,4.9725,-0.0169,-0.0595,0.0102,313.3219,302.2305,321.4363,0.0844,0.0351,0.1558,0.0525,0.0218,0.0971,0.0365,0.0152,0.0672,"MARCS"
1636148068921376768,"Gaia DR3 2512556873600",2512556873600,831324186,2016.0,45.07492471994457,33.54464,0.1108820420939314,24.830378,null,null,null,null,null,null,null,null,-0.1309745,null,null,null,null,null,null,null,null,null,47,0,47,0,35.859825,1964.6633,119.27316,184.45139,3,"False",null,null,null,null,null,null,null,null,7,6,120.0138,8,8,0,0.41287413,54.55582,0,0,null,0.4272169,0.21570276,0.5622911,0.8563249,-75.8122,-1.6494459,-50.740902,30.197552,"False",47,45.0438638293822,2.318626,19.426964,21.553278,5,296.537445766598,54.37354,5.4537086,19.158344,5,444.69570491458535,83.88305,5.3013773,18.127739,16.455807,0,0,0,0,0,1.0306053,-2.3949337,3.425539,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"NOT_AVAILABLE",176.91896393240253,-48.771755887594544,42.64430938681278,-16.251990466593394,"False","False",0,"False","False","False","False","False","False","False","False",0.004575429,0.036596417,0.9587768,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null
1636148068921376768,"Gaia DR3 2821794351744",2821794351744,1389218613,2016.0,45.124152453120615,1.6364167,0.13677852527701542,1.5161866,null,null,null,null,null,null,null,null,-0.057091914,null,null,null,null,null,null,null,null,null,162,0,145,17,85.27109,11890.783,11.945554,971.3325,3,"False",null,null,null,null,null,null,null,null,18,12,3.0579515,20,5,0,0.17080484,85.50274,58,0,null,0.3759727,0.1906163,0.49689272,0.8553549,-98.59949,-1.5183473,-46.745052,28.801044,"False",162,1180.830086047288,13.438973,87.86609,18.006899,18,448.9534667774457,10.775265,41.66519,18.708038,16,2295.837741126928,23.04338,99.63112,16.345543,2.324459,0,2,0,1,0,2.3624954,0.70113945,1.661356,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"NOT_AVAILABLE",176.94249514790158,-48.718340373548315,42.70123151869663,-16.2416233995086,"False","False",0,"False","False","False","False","False","False","False","False",0.0,0.0,0.9105993,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null
$\endgroup$
9
  • $\begingroup$ Is this an astronomy question? $\endgroup$
    – ProfRob
    Commented Aug 18, 2022 at 17:44
  • $\begingroup$ At least astronomy related for sure. I honestly don't know where else to post such a question. Any tips from your side? $\endgroup$ Commented Aug 18, 2022 at 17:56
  • $\begingroup$ Since the file is approximately double the size, I'd say the ecsv version is using Unicode, where each character takes two bytes, rather than ASCII where each char takes just one byte. $\endgroup$ Commented Aug 18, 2022 at 18:10
  • $\begingroup$ From what I know, FITS is able to store data in binary format, which considerably reduces the size. Regarding CSV and ECSV, it depends on how you store the data. In the simplest case, a CSV file (with a proper header) should be a valid ECSV. In that case they would have the same size. But ECSV can also do some fancy things with masks and arrays and objects, and using those increases the size. Could you show an extract of the CSV and the ECSV? $\endgroup$
    – Prallax
    Commented Aug 18, 2022 at 18:55
  • $\begingroup$ @GregMiller this points me in the right direction. Thanks! $\endgroup$ Commented Aug 18, 2022 at 20:01

1 Answer 1

2
$\begingroup$

FITS is a versatile format that may contain binary data. Now, storing numbers in binary rather than in ascii is a great advantage in terms of disk space.

Consider for instance a double floating point number, which is represented in binary with 8 bytes. The same number, if written with all significant digits in ascii, will be about 16-19 characters, depending on the sign and how many digits one considers significant. Since each ascii character is 1 byte, this means that the binary format occupies about half of the space. A similar argument can be made for integers and things get even better with booleans, which in binary are just 1 bit, versus the 5 bytes of "false" and the 4 bytes of "true".

FITS binary tables consist of usually short headers of key, value, comment which are 80 bytes each followed by the binary table itself which is just the size of the data types. So if I have a table of 2 doubles (8 bytes each), 4 floats (4 bytes each) and 1 double value per row, then the space used for that row is just the sum of the sizes (40 bytes), no padding at all. FITS sections (headers and tables/images) are blocked into 2880 byte blocks so a 10 line header uses 2880 bytes not 10*80=800 bytes

Therefore my guess would be that at least part of the data in the FITS file is in binary format.

Regarding the csv and the ecsv formats, they are similar, but ecsv is a bit more strict on the syntax and allows to store also multidimensional data and unstructured objects. These fancy capabilities of course come at a price, which is larger space on disk, but it is not the case here: the data being stored is the same, it is just a table.

The difference becomes apparent when looking at the extracts of the files. They are almost identical, but for the treatment of empty fields. In the csv file, empty fields are just left empty, no characters between the commas. The ecsv writes "null" instead.

In the 10 lines of the extract, there are 351 nulls, which contain about 11% of the total number of characters. Since the difference in size between the csv and the ecsv files is about 18%, it is reasonable to assume that the presence of nulls explains most of this difference.

$\endgroup$
3
  • 1
    $\begingroup$ Yes, FITS binary tables consist of usually short headers of key, value, comment which are 80 bytes each followed by the binary table itself which is just the size of the data types. So if I have a table of 2 doubles (8 bytes each), 4 floats (4 bytes each) and 1 double value per row, then the space used for that row is just the sum of the sizes (40 bytes), no padding at all. FITS sections (headers and tables/images) are blocked into 2880 byte blocks so a 10 line header uses 2880 bytes not 10*80=800 bytes $\endgroup$ Commented Aug 18, 2022 at 23:29
  • $\begingroup$ Thank you, I will incorporate this into the answer $\endgroup$
    – Prallax
    Commented Aug 19, 2022 at 6:47
  • $\begingroup$ You couldn't wish for a more detailed answer. Thanks both of you so much! $\endgroup$ Commented Aug 19, 2022 at 9:32

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .