The continuous improvement of the performance of optical clocks as well as of laser ranging interferometry in space with LISA Pathfinder and GRACE-FO has the potential to set up a network of optical clocks in space as well as a swarm of satellites in which relative distances and frequencies are continuously monitored. Both, separately or together, they will enable a global determination of the gravitational field of the Earth with very high precision. In order to be able to consistently interpret the data a fully general relativistic description has to be employed. Furthermore, the formalism for a general relativistic geodesy has to be developed further to include non-stationary observers, in particular with moving clocks. From such measurements the gravitoelectric and gravitomagnetic potential of the Earth together with the spatial metric components shall be determined. Also a multipole expansion of all metrical components will be introduced, see also C02, and linked to the above mentioned observables. The relevance of the relativistic contributions will be estimated. Based on sophisticated simulations, where we will consider realistic error assumptions, we will study and quantify the contribution of space-based clocks for gravity field recovery. Finally, possible new tests of General Relativity (GR) will be proposed and, if possible, performed. For the longer perspective the potential of novel technologies like entangled clocks or composite clocks in space will be analysed.