remind issueshttps://gitlab.pik-potsdam.de/hilaire/remind/-/issues2021-01-04T14:21:30+01:00https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/274REMIND Run Portability Snags2021-01-04T14:21:30+01:00Michaja PehlREMIND Run Portability SnagsThe idea used to be that a REMIND run can be used from its directory independent of all other REMIND code. Currently, that's not the case any more in at least two instances. I copied a run to a temporary directory for debugging and got...The idea used to be that a REMIND run can be used from its directory independent of all other REMIND code. Currently, that's not the case any more in at least two instances. I copied a run to a temporary directory for debugging and got this:
```
$ grep Error log.txt | sort | uniq
Error in fread("../../modules/35_transport/edge_esm/input/EDGEscenario_description.csv") :
Error in read.magpie("../../modules/11_aerosols/exoGAINS/input/ef_gains.cs4r")
```
So at least two files need to be included in https://github.com/remindmodel/remind/blob/be7a866d4318ce78cd09a92b7f9637421dd3f106/config/default.cfg#L460-L475 and scripts need to be adjusted.https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/286geohdr infeasibilities2020-12-03T17:05:55+01:00Michaja Pehlgeohdr infeasibilities**tl;dr:** EDGE-Transport needs to stop messing with the past like a naughty time traveller. ;)
I'm pretty confident I found the origin of the infeasibilities having to do with `geohdr` capacity in the first free time step.
Ultimat...**tl;dr:** EDGE-Transport needs to stop messing with the past like a naughty time traveller. ;)
I'm pretty confident I found the origin of the infeasibilities having to do with `geohdr` capacity in the first free time step.
Ultimately, it is caused by `EDGE_transport.R` modifying the values of `p35_fe2es`
https://github.com/remindmodel/remind/blob/547ba8f7be1bfc8adbd9cf1c39438089e9c0b174/modules/35_transport/edge_esm/presolve.gms#L13-L19
These end up in the `fulldata.gdx`, which becomes the `input_ref.gdx` for the next run, where they are loaded, overwriting the data from `./modules/35_transport/edge_esm/input/fe2es.cs4r`.
https://github.com/remindmodel/remind/blob/547ba8f7be1bfc8adbd9cf1c39438089e9c0b174/modules/35_transport/edge_esm/datainput.gms#L50-L54
Then, different `pm_fe2es` values change the result of `initialcap2`, specifically `v05_INIcap0`/`pm_cap0`, which trickles down to change the vintage capital structure of `geohdr` through `pm_cap0` → `s05_aux_tot_prod` → `s05_aux_prod_remaining` → `p05_aux_prod_thisgrade` → `p05_aux_cap_distr` → `p05_aux_cap` → `sm_tmp` → `vm_capDistr.fx`:
https://github.com/remindmodel/remind/blob/547ba8f7be1bfc8adbd9cf1c39438089e9c0b174/modules/05_initialCap/on/preloop.gms#L145-L182
Then, depending on region and scenario settings, it is possible that (a) CONOPT is in a weird spot it can't find out of (my Npi run) or (b) the modified vintage capacity is so high that _capacity × capacity factor_ exceeds `pm_dataren(regi,"maxprod",rlf,te)` (Felix` NSW run) – which isn't a problem for the fixed time-steps, as `q_limitProd` is [defined only over the free ones](https://github.com/remindmodel/remind/blob/547ba8f7be1bfc8adbd9cf1c39438089e9c0b174/core/equations.gms#L400-L407).
@LaviniaBaumstark @dklein-pik @giannou (In lieu of a RSE github team):
I think it's problematic that a run, that is supposed to be fixed on a different run up to some point, has a different vintage structure. I was always puzzled by the fact that `initialcap2` is run for _every run_. This seems not only to be the case for `geohdr`, but other technologies too. Small thing that doesn't show up in the results (`vm_cap` is still the same), but this bug wouldn't happened without `initialcap2` running again, although its results should be the same in any case.
In any case, preserving the 2005 values of `pm_fe2es` at the `./modules/35_transport/edge_esm/input/fe2es.cs4r` levels should fix this.
-----
Evidence/notes using the failed DeepEl3 Npi run as an example: `/p/tmp/pehl/Remind_DeepEl/output/TraInd-Npi_2020-11-27_16.06.51`
```
$ gdxdiff input_ref.gdx non_optimal.gdx id=pm_cap0,pm_EN_demand_from_initialcap2,pm_cesdata,pm_shFeCes,pm_fe2es,pm_cf,pm_data,pm_prodCouple,p35_fe2es,EDGE_scenario
GDXDIFF 31.1.1 r4b06116 Released May 16, 2020 LEG x86 64bit/Linux
File1 : input_ref.gdx
File2 : non_optimal.gdx
Id : pm_cap0 pm_EN_demand_from_initialcap2 pm_cesdata pm_shFeCes pm_fe2es pm_cf pm_data pm_prodCouple p35_fe2es EDGE_scenario
Summary of differences:
pm_cap0 Data are different
pm_cf Data are different
pm_data Data are different
pm_EN_demand_from_initialcap2 Data are different
pm_fe2es Data are different
pm_shFeCes Data are different
Output: diffile.gdx
GDXDiff finished
```
```
$ dumpgdx diffile.gdx vm_deltaCap "CHA.*geohdr.*L\>" | head
1995 CHA geohdr 1 dif1 L 8.2494690248504E-8
1995 CHA geohdr 1 dif2 L 8.24965476551363E-8
2000 CHA geohdr 1 dif1 L 1.42715686791135E-6
2000 CHA geohdr 1 dif2 L 1.42718900101742E-6
2015 CHA geohdr 1 dif1 L 0.000906719440325381
2015 CHA geohdr 1 dif2 L 0.000906719440325381
2020 CHA geohdr 1 dif1 L 3.4181901481737E-5
2020 CHA geohdr 1 dif2 L 3.4181901481737E-5
2030 CHA geohdr 1 dif1 L 0.000100553100549173
2030 CHA geohdr 1 dif2 L 0.000100553067898506
```
```
$ dumpgdx diffile.gdx pm_cap0 "CHA.*geohdr"
CHA geohdr dif1 1.49901612654113E-5
CHA geohdr dif2 1.4990498775921E-5
```
```
$ dumpgdx diffile.gdx pm_fe2es "2005.*CHA"
2005 CHA te_espet_pass_sm dif1 32.7810653744073
2005 CHA te_espet_pass_sm dif2 32.7803397536317
2005 CHA te_esdie_pass_sm dif1 72.8208842393301
2005 CHA te_esdie_pass_sm dif2 72.6703864713621
2005 CHA te_eselt_pass_sm dif1 274.077206442013
2005 CHA te_eselt_pass_sm dif2 276.29708722215
2005 CHA te_esh2t_pass_sm dif1 64.7583317381968
2005 CHA te_esh2t_pass_sm dif2 64.742052256529
2005 CHA te_esgat_pass_sm dif1 66.8262951982073
2005 CHA te_esgat_pass_sm dif2 67.1300018776568
2005 CHA te_esdie_frgt_sm dif1 35.041588297353
2005 CHA te_esdie_frgt_sm dif2 34.9779610388119
2005 CHA te_eselt_frgt_sm dif1 226.082090011469
2005 CHA te_eselt_frgt_sm dif2 226.082090011469
2005 CHA te_esh2t_frgt_sm dif1 33.3906087857954
2005 CHA te_esh2t_frgt_sm dif2 33.391690768669
2005 CHA te_esgat_frgt_sm dif1 13.2919587450842
2005 CHA te_esgat_frgt_sm dif2 13.3447480682688
```https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/265iteration/resource limits on CONOPT2020-10-27T13:04:08+01:00Michaja Pehliteration/resource limits on CONOPThttps://github.com/remindmodel/remind/blob/3dc35199bbf5544b148d2e9e2f706761220cec63/core/loop.gms#L18-L21
Could we release these limits on CONOPT?
I'm mainly interested in `reslim` for Nash, because the REMIND-EU calibration has sh...https://github.com/remindmodel/remind/blob/3dc35199bbf5544b148d2e9e2f706761220cec63/core/loop.gms#L18-L21
Could we release these limits on CONOPT?
I'm mainly interested in `reslim` for Nash, because the REMIND-EU calibration has shown to need more time to find a feasible solution (at least three and a half hours). I don't think limiting `reslim` is useful (anymore).
- Most runs solve faster, so this limit isn't reached very often (aka does not matter).
- If CONOPT does not find a solution, I would like it to keep trying. It is likely to stop at some point because the gradient gets too flat or it hits a `NA`. _But if it doesn't_, having it keep going may give me a feasible (unconverged) .gdx from which to start a new run. (Worked for me a lot for initial calibrations of new parametrisations.)
- When CONOPT stops because of a time-out, it is not likely to find a feasible solution within two hours in the next `solve` iteration or even Nash iteration, but will just waste eleven iterations trying again from the beginning, without informing the user who could then intervene.
- Runs are limited by slurm in any case, so we don't run the risk of wasting more cluster resources if this is dropped.
The non-Nash `reslim` (eleven days) isn't really different from the default (317 years) – most runs will get cancelled well before, and if not (Negishi?), users are likely to check in on them.
Same argument for `iterlim`. 1e6 is a random number that is not qualitatively different from the default (2e9).https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/189make the run status (rs) script work for coupled runs also2020-09-18T11:54:10+02:00Jerome Hilairemake the run status (rs) script work for coupled runs also*Created by: giannou*
*Created by: giannou*
https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/193Short-term issues2020-06-19T12:26:22+02:00Jerome HilaireShort-term issues*Created by: amnmalik*
For Baseline scenarios -
- Near-term trend for SE|Gases|Coal (GLO) doesn't match to bottom-up data.
- Near-term trend for SE|Solids|Coal (GLO) doesn't match to bottom-up data.
- For SE|Solids|Coal, historical v...*Created by: amnmalik*
For Baseline scenarios -
- Near-term trend for SE|Gases|Coal (GLO) doesn't match to bottom-up data.
- Near-term trend for SE|Solids|Coal (GLO) doesn't match to bottom-up data.
- For SE|Solids|Coal, historical values not matched for JPN, REF.
- For SE|Solids|Coal, near-term trends don't match for CHA.
For SDP scenarios -
- PE|Coal - GLO, CHA, NEU, SSA doesn't match short-term trends.
- Emi|CO2|Energy|Supply|Electricity for LAM doesn't match recent trends.
- Emi|CO2|Energy|Supply|Electricity, historical mismatch for EUR
- Emi|CO2|Gross Fossil Fuels and Industry, recent trends not matched for NEU
- calibration USA feso -A spike in demand around 2020 (FE|Buildings|Solids)
General -
- SE Gases from biomass not in line with historical data (CAZ, CHA, EUR, LAM, MEA, OAS)
- SE Gases from coal totally off in CHA, IND, REF, SSA, OAS
- Less PE Gas in CAZ and REF compared with historical data (2010-2015)
- SSP2 - NDC - Consumption losses different across scenarios before 2020
- SE Electricity from gas not in line with historical data in CAZ (also IND, NEU, REF to a much smaller extent)
- High policy costs for CAZ under SSP2 NDC
https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/181How is the LCOEplot reporting supposed to be used?2020-06-01T14:18:51+02:00Jerome HilaireHow is the LCOEplot reporting supposed to be used?*Created by: giannou*
https://github.com/remindmodel/remind/blob/develop/scripts/output/single/LCOEPlot.R
throws an error when called via "Rscript output.R"
*Created by: giannou*
https://github.com/remindmodel/remind/blob/develop/scripts/output/single/LCOEPlot.R
throws an error when called via "Rscript output.R"
https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/146SSP2-PkBudg1100 runs with latest REMIND release crash with division by zero2020-05-13T17:18:30+02:00Jerome HilaireSSP2-PkBudg1100 runs with latest REMIND release crash with division by zero*Created by: giannou*
/home/giannou/REMIND/output/SSP2-PkBudg1100_2020-04-13_16.10.08/full.lst*Created by: giannou*
/home/giannou/REMIND/output/SSP2-PkBudg1100_2020-04-13_16.10.08/full.lsthttps://gitlab.pik-potsdam.de/hilaire/remind/-/issues/170full.lst lagging considerably behind full.log2020-05-12T14:35:26+02:00Jerome Hilairefull.lst lagging considerably behind full.log*Created by: giannou*
Why is it that full.log reports being at the 12th iteration whereas full.lst hasn't even started reporting the first one? Does anyone have an idea?*Created by: giannou*
Why is it that full.log reports being at the 12th iteration whereas full.lst hasn't even started reporting the first one? Does anyone have an idea?Michaja PehlMichaja Pehlhttps://gitlab.pik-potsdam.de/hilaire/remind/-/issues/165Current default is non-optimal2020-05-07T16:00:48+02:00Jerome HilaireCurrent default is non-optimal*Created by: johanneskoch94*
The current default version is non-optimal. Is there some work-in-progress going on?*Created by: johanneskoch94*
The current default version is non-optimal. Is there some work-in-progress going on?https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/113start.R --restart does not submit slurm jobs, runs on login node instead2020-04-30T12:03:43+02:00Michaja Pehlstart.R --restart does not submit slurm jobs, runs on login node insteadhttps://gitlab.pik-potsdam.de/hilaire/remind/-/issues/129git version and changes written somewhere in the run?2020-04-23T17:55:48+02:00Jerome Hilairegit version and changes written somewhere in the run?*Created by: giannou*
We used to have the SVN/git version and the changes with respect to these reported in the run files, is this not the case any more? It was a very helpful feature.*Created by: giannou*
We used to have the SVN/git version and the changes with respect to these reported in the run files, is this not the case any more? It was a very helpful feature.https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/139How can the SLURM parameter '#SBATCH --time' be adjusted?2020-04-23T13:25:04+02:00Jerome HilaireHow can the SLURM parameter '#SBATCH --time' be adjusted?*Created by: giannou*
*Created by: giannou*
https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/140How can a run be restarted, that was interrupted before a fulldata.gdx has be...2020-04-22T17:05:37+02:00Jerome HilaireHow can a run be restarted, that was interrupted before a fulldata.gdx has been created?*Created by: giannou*
*Created by: giannou*
https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/117Usage of start.R is not documented2020-04-14T09:47:31+02:00Jerome HilaireUsage of start.R is not documented*Created by: Loisel*
Please indicate at the top of the file how it is used and which parameters or flags can be given.
Alternatively, one could use `optparse` and provide `--help` or `--usage`.*Created by: Loisel*
Please indicate at the top of the file how it is used and which parameters or flags can be given.
Alternatively, one could use `optparse` and provide `--help` or `--usage`.https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/112start.R fails to restart fixed runs2020-04-07T16:24:56+02:00Michaja Pehlstart.R fails to restart fixed runsby not decompressing `levs.gms.gz`, `margs.gms.gz`, and `fixings.gms.gz`.by not decompressing `levs.gms.gz`, `margs.gms.gz`, and `fixings.gms.gz`.https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/134current version is broken2020-04-07T13:12:00+02:00Jerome Hilairecurrent version is broken*Created by: giannou*
out-of-the-box REMIND run with module piam/1.10 loaded fails with error message:
Error in setNames(pop[regi, , ] * log(1000 * cons[regi, , ] * (1 - (c_damage * :
dims [product 19] do not match the length of ...*Created by: giannou*
out-of-the-box REMIND run with module piam/1.10 loaded fails with error message:
Error in setNames(pop[regi, , ] * log(1000 * cons[regi, , ] * (1 - (c_damage * :
dims [product 19] do not match the length of object [0]
Calls: mbind -> reportMacroEconomy -> setNames
after the 1st iteration, when running exoGAINSAirpollutants.Rhttps://gitlab.pik-potsdam.de/hilaire/remind/-/issues/86asked for Negishi, scripts start Nash2020-03-02T15:00:37+01:00Jerome Hilaireasked for Negishi, scripts start Nash*Created by: giannou*
![Screenshot 2020-03-02 12 41 33](https://user-images.githubusercontent.com/11047746/75673593-9f38ea80-5c83-11ea-9407-1b4608b6e92b.png)
*Created by: giannou*
![Screenshot 2020-03-02 12 41 33](https://user-images.githubusercontent.com/11047746/75673593-9f38ea80-5c83-11ea-9407-1b4608b6e92b.png)
https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/56document calculation of costs from MAC curves2020-02-11T15:59:13+01:00Michaja Pehldocument calculation of costs from MAC curvesMichaja PehlMichaja Pehlhttps://gitlab.pik-potsdam.de/hilaire/remind/-/issues/53v_co2capturevalve is venting industry CCS2020-02-05T11:44:28+01:00Michaja Pehlv_co2capturevalve is venting industry CCShttps://github.com/remindmodel/remind/blob/c67cc61e1ef4d6a36c7dd487bc58e22f51294ae8/core/equations.gms#L696-L702
c.f. `default_2020-01-08_14.13.34` run, 2020/USA
```
vm_co2capture 2020 USA cco2 ico2 ccsinje 1 L 0.0030434...https://github.com/remindmodel/remind/blob/c67cc61e1ef4d6a36c7dd487bc58e22f51294ae8/core/equations.gms#L696-L702
c.f. `default_2020-01-08_14.13.34` run, 2020/USA
```
vm_co2capture 2020 USA cco2 ico2 ccsinje 1 L 0.00304342219022761
v_co2capturevalve 2020 USA L 0.00194586349440526
vm_co2CCS 2020 USA cco2 ico2 ccsinje 1 L 0.00109755869582235
vm_emiTeDetail 2020 USA * * * cco2 L 0.00036196011504294
vm_ccs_cdr 2020 USA cco2 ico2 ccsinje * L 0
vm_emiIndCCS 2020 USA * L 0.00268146207518457
```
`vm_co2capture` = `vm_co2CCS` + `v_co2capturevalve` and
`vm_co2CCS` = `vm_emiTeDetail` + `vm_ccs_cdr` + `vm_emiIndCCS`
So CO2 captured in industry gets vented. I'm not entirely sure what the purpose of `v_co2capturevalve` is
https://github.com/remindmodel/remind/blob/c67cc61e1ef4d6a36c7dd487bc58e22f51294ae8/core/declarations.gms#L293
but if industry is supposed to supply CO2 for CCU, that should be modelled explicitly.https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/54add explicit reporting of biomass in industry FE use2020-02-04T10:40:53+01:00Michaja Pehladd explicit reporting of biomass in industry FE useMichaja PehlMichaja Pehl