remind issueshttps://gitlab.pik-potsdam.de/hilaire/remind/-/issues2021-02-10T15:47:22+01:00https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/299Download and distribute input data error2021-02-10T15:47:22+01:00Jerome HilaireDownload and distribute input data error*Created by: fei-cheng*
Hi,
I downloaded the latest release and tried to run this model. It threw an error while trying to download and distribute input data. The error message looks like:
```
Load data..
Following files not found...*Created by: fei-cheng*
Hi,
I downloaded the latest release and tried to run this model. It threw an error while trying to download and distribute input data. The error message looks like:
```
Load data..
Following files not found:
rev5.961_690d3718e151be1b450b394c1064b1c5_remind.tgz
```
My understanding is it tries to download the above tgz file from somewhere. But where is the file stored?
And according to the README, the input data is not part of this project. But do we provide any example data for me to understand the structure and the format of the input data?
Many thanks!
https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/302reduce github repository size2021-02-10T14:06:05+01:00Jerome Hilairereduce github repository size*Created by: giannou*
*Created by: giannou*
https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/286geohdr infeasibilities2020-12-03T17:05:55+01:00Michaja Pehlgeohdr infeasibilities**tl;dr:** EDGE-Transport needs to stop messing with the past like a naughty time traveller. ;)
I'm pretty confident I found the origin of the infeasibilities having to do with `geohdr` capacity in the first free time step.
Ultimat...**tl;dr:** EDGE-Transport needs to stop messing with the past like a naughty time traveller. ;)
I'm pretty confident I found the origin of the infeasibilities having to do with `geohdr` capacity in the first free time step.
Ultimately, it is caused by `EDGE_transport.R` modifying the values of `p35_fe2es`
https://github.com/remindmodel/remind/blob/547ba8f7be1bfc8adbd9cf1c39438089e9c0b174/modules/35_transport/edge_esm/presolve.gms#L13-L19
These end up in the `fulldata.gdx`, which becomes the `input_ref.gdx` for the next run, where they are loaded, overwriting the data from `./modules/35_transport/edge_esm/input/fe2es.cs4r`.
https://github.com/remindmodel/remind/blob/547ba8f7be1bfc8adbd9cf1c39438089e9c0b174/modules/35_transport/edge_esm/datainput.gms#L50-L54
Then, different `pm_fe2es` values change the result of `initialcap2`, specifically `v05_INIcap0`/`pm_cap0`, which trickles down to change the vintage capital structure of `geohdr` through `pm_cap0` → `s05_aux_tot_prod` → `s05_aux_prod_remaining` → `p05_aux_prod_thisgrade` → `p05_aux_cap_distr` → `p05_aux_cap` → `sm_tmp` → `vm_capDistr.fx`:
https://github.com/remindmodel/remind/blob/547ba8f7be1bfc8adbd9cf1c39438089e9c0b174/modules/05_initialCap/on/preloop.gms#L145-L182
Then, depending on region and scenario settings, it is possible that (a) CONOPT is in a weird spot it can't find out of (my Npi run) or (b) the modified vintage capacity is so high that _capacity × capacity factor_ exceeds `pm_dataren(regi,"maxprod",rlf,te)` (Felix` NSW run) – which isn't a problem for the fixed time-steps, as `q_limitProd` is [defined only over the free ones](https://github.com/remindmodel/remind/blob/547ba8f7be1bfc8adbd9cf1c39438089e9c0b174/core/equations.gms#L400-L407).
@LaviniaBaumstark @dklein-pik @giannou (In lieu of a RSE github team):
I think it's problematic that a run, that is supposed to be fixed on a different run up to some point, has a different vintage structure. I was always puzzled by the fact that `initialcap2` is run for _every run_. This seems not only to be the case for `geohdr`, but other technologies too. Small thing that doesn't show up in the results (`vm_cap` is still the same), but this bug wouldn't happened without `initialcap2` running again, although its results should be the same in any case.
In any case, preserving the 2005 values of `pm_fe2es` at the `./modules/35_transport/edge_esm/input/fe2es.cs4r` levels should fix this.
-----
Evidence/notes using the failed DeepEl3 Npi run as an example: `/p/tmp/pehl/Remind_DeepEl/output/TraInd-Npi_2020-11-27_16.06.51`
```
$ gdxdiff input_ref.gdx non_optimal.gdx id=pm_cap0,pm_EN_demand_from_initialcap2,pm_cesdata,pm_shFeCes,pm_fe2es,pm_cf,pm_data,pm_prodCouple,p35_fe2es,EDGE_scenario
GDXDIFF 31.1.1 r4b06116 Released May 16, 2020 LEG x86 64bit/Linux
File1 : input_ref.gdx
File2 : non_optimal.gdx
Id : pm_cap0 pm_EN_demand_from_initialcap2 pm_cesdata pm_shFeCes pm_fe2es pm_cf pm_data pm_prodCouple p35_fe2es EDGE_scenario
Summary of differences:
pm_cap0 Data are different
pm_cf Data are different
pm_data Data are different
pm_EN_demand_from_initialcap2 Data are different
pm_fe2es Data are different
pm_shFeCes Data are different
Output: diffile.gdx
GDXDiff finished
```
```
$ dumpgdx diffile.gdx vm_deltaCap "CHA.*geohdr.*L\>" | head
1995 CHA geohdr 1 dif1 L 8.2494690248504E-8
1995 CHA geohdr 1 dif2 L 8.24965476551363E-8
2000 CHA geohdr 1 dif1 L 1.42715686791135E-6
2000 CHA geohdr 1 dif2 L 1.42718900101742E-6
2015 CHA geohdr 1 dif1 L 0.000906719440325381
2015 CHA geohdr 1 dif2 L 0.000906719440325381
2020 CHA geohdr 1 dif1 L 3.4181901481737E-5
2020 CHA geohdr 1 dif2 L 3.4181901481737E-5
2030 CHA geohdr 1 dif1 L 0.000100553100549173
2030 CHA geohdr 1 dif2 L 0.000100553067898506
```
```
$ dumpgdx diffile.gdx pm_cap0 "CHA.*geohdr"
CHA geohdr dif1 1.49901612654113E-5
CHA geohdr dif2 1.4990498775921E-5
```
```
$ dumpgdx diffile.gdx pm_fe2es "2005.*CHA"
2005 CHA te_espet_pass_sm dif1 32.7810653744073
2005 CHA te_espet_pass_sm dif2 32.7803397536317
2005 CHA te_esdie_pass_sm dif1 72.8208842393301
2005 CHA te_esdie_pass_sm dif2 72.6703864713621
2005 CHA te_eselt_pass_sm dif1 274.077206442013
2005 CHA te_eselt_pass_sm dif2 276.29708722215
2005 CHA te_esh2t_pass_sm dif1 64.7583317381968
2005 CHA te_esh2t_pass_sm dif2 64.742052256529
2005 CHA te_esgat_pass_sm dif1 66.8262951982073
2005 CHA te_esgat_pass_sm dif2 67.1300018776568
2005 CHA te_esdie_frgt_sm dif1 35.041588297353
2005 CHA te_esdie_frgt_sm dif2 34.9779610388119
2005 CHA te_eselt_frgt_sm dif1 226.082090011469
2005 CHA te_eselt_frgt_sm dif2 226.082090011469
2005 CHA te_esh2t_frgt_sm dif1 33.3906087857954
2005 CHA te_esh2t_frgt_sm dif2 33.391690768669
2005 CHA te_esgat_frgt_sm dif1 13.2919587450842
2005 CHA te_esgat_frgt_sm dif2 13.3447480682688
```https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/252dependency problem on Mac: regionscode function not found2020-10-06T13:49:27+02:00Jerome Hilairedependency problem on Mac: regionscode function not found*Created by: jeh0753*
Installing the latest R version, GAMS version, and following the documentation through on Mac brings the following error when running `source('start.R')`:
<img width="567" alt="image" src="https://user-images.g...*Created by: jeh0753*
Installing the latest R version, GAMS version, and following the documentation through on Mac brings the following error when running `source('start.R')`:
<img width="567" alt="image" src="https://user-images.githubusercontent.com/1597013/94822730-fac18700-03d0-11eb-8679-696b6fb23b20.png">
I've recreated this on two separate Mac machines. If I'm able to sort it out I'll post here, but any ideas or solutions would be a great help.https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/274REMIND Run Portability Snags2021-01-04T14:21:30+01:00Michaja PehlREMIND Run Portability SnagsThe idea used to be that a REMIND run can be used from its directory independent of all other REMIND code. Currently, that's not the case any more in at least two instances. I copied a run to a temporary directory for debugging and got...The idea used to be that a REMIND run can be used from its directory independent of all other REMIND code. Currently, that's not the case any more in at least two instances. I copied a run to a temporary directory for debugging and got this:
```
$ grep Error log.txt | sort | uniq
Error in fread("../../modules/35_transport/edge_esm/input/EDGEscenario_description.csv") :
Error in read.magpie("../../modules/11_aerosols/exoGAINS/input/ef_gains.cs4r")
```
So at least two files need to be included in https://github.com/remindmodel/remind/blob/be7a866d4318ce78cd09a92b7f9637421dd3f106/config/default.cfg#L460-L475 and scripts need to be adjusted.https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/265iteration/resource limits on CONOPT2020-10-27T13:04:08+01:00Michaja Pehliteration/resource limits on CONOPThttps://github.com/remindmodel/remind/blob/3dc35199bbf5544b148d2e9e2f706761220cec63/core/loop.gms#L18-L21
Could we release these limits on CONOPT?
I'm mainly interested in `reslim` for Nash, because the REMIND-EU calibration has sh...https://github.com/remindmodel/remind/blob/3dc35199bbf5544b148d2e9e2f706761220cec63/core/loop.gms#L18-L21
Could we release these limits on CONOPT?
I'm mainly interested in `reslim` for Nash, because the REMIND-EU calibration has shown to need more time to find a feasible solution (at least three and a half hours). I don't think limiting `reslim` is useful (anymore).
- Most runs solve faster, so this limit isn't reached very often (aka does not matter).
- If CONOPT does not find a solution, I would like it to keep trying. It is likely to stop at some point because the gradient gets too flat or it hits a `NA`. _But if it doesn't_, having it keep going may give me a feasible (unconverged) .gdx from which to start a new run. (Worked for me a lot for initial calibrations of new parametrisations.)
- When CONOPT stops because of a time-out, it is not likely to find a feasible solution within two hours in the next `solve` iteration or even Nash iteration, but will just waste eleven iterations trying again from the beginning, without informing the user who could then intervene.
- Runs are limited by slurm in any case, so we don't run the risk of wasting more cluster resources if this is dropped.
The non-Nash `reslim` (eleven days) isn't really different from the default (317 years) – most runs will get cancelled well before, and if not (Negishi?), users are likely to check in on them.
Same argument for `iterlim`. 1e6 is a random number that is not qualitatively different from the default (2e9).https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/239Document values in generisdata2020-09-16T12:59:37+02:00Jerome HilaireDocument values in generisdata*Created by: Loisel*
There should be a facility to document values in generisdata, as REMIND results are quite sensitive to these assumptions. One possibility would be to start from a YAML file and generate the `.prn`, thereby ensuring ...*Created by: Loisel*
There should be a facility to document values in generisdata, as REMIND results are quite sensitive to these assumptions. One possibility would be to start from a YAML file and generate the `.prn`, thereby ensuring the presence of some documentation string (ideally containing a reference).https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/223filter out files in run status (rs), keep only directories2020-08-26T14:56:02+02:00Jerome Hilairefilter out files in run status (rs), keep only directories*Created by: giannou*
*Created by: giannou*
https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/222run status (rs) script reports wrong results in debug mode2021-02-10T14:10:46+01:00Jerome Hilairerun status (rs) script reports wrong results in debug mode*Created by: giannou*
*Created by: giannou*
https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/193Short-term issues2020-06-19T12:26:22+02:00Jerome HilaireShort-term issues*Created by: amnmalik*
For Baseline scenarios -
- Near-term trend for SE|Gases|Coal (GLO) doesn't match to bottom-up data.
- Near-term trend for SE|Solids|Coal (GLO) doesn't match to bottom-up data.
- For SE|Solids|Coal, historical v...*Created by: amnmalik*
For Baseline scenarios -
- Near-term trend for SE|Gases|Coal (GLO) doesn't match to bottom-up data.
- Near-term trend for SE|Solids|Coal (GLO) doesn't match to bottom-up data.
- For SE|Solids|Coal, historical values not matched for JPN, REF.
- For SE|Solids|Coal, near-term trends don't match for CHA.
For SDP scenarios -
- PE|Coal - GLO, CHA, NEU, SSA doesn't match short-term trends.
- Emi|CO2|Energy|Supply|Electricity for LAM doesn't match recent trends.
- Emi|CO2|Energy|Supply|Electricity, historical mismatch for EUR
- Emi|CO2|Gross Fossil Fuels and Industry, recent trends not matched for NEU
- calibration USA feso -A spike in demand around 2020 (FE|Buildings|Solids)
General -
- SE Gases from biomass not in line with historical data (CAZ, CHA, EUR, LAM, MEA, OAS)
- SE Gases from coal totally off in CHA, IND, REF, SSA, OAS
- Less PE Gas in CAZ and REF compared with historical data (2010-2015)
- SSP2 - NDC - Consumption losses different across scenarios before 2020
- SE Electricity from gas not in line with historical data in CAZ (also IND, NEU, REF to a much smaller extent)
- High policy costs for CAZ under SSP2 NDC
https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/189make the run status (rs) script work for coupled runs also2020-09-18T11:54:10+02:00Jerome Hilairemake the run status (rs) script work for coupled runs also*Created by: giannou*
*Created by: giannou*
https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/181How is the LCOEplot reporting supposed to be used?2020-06-01T14:18:51+02:00Jerome HilaireHow is the LCOEplot reporting supposed to be used?*Created by: giannou*
https://github.com/remindmodel/remind/blob/develop/scripts/output/single/LCOEPlot.R
throws an error when called via "Rscript output.R"
*Created by: giannou*
https://github.com/remindmodel/remind/blob/develop/scripts/output/single/LCOEPlot.R
throws an error when called via "Rscript output.R"
https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/180Reporting error: Fortran runtime Error in the Magicc Reporting2020-10-26T13:41:21+01:00Jerome HilaireReporting error: Fortran runtime Error in the Magicc Reporting*Created by: Loisel*
When `reporting.R` is executed, the magicc reporting parts fail with
```
[1] "Executing reporting.R" ...*Created by: Loisel*
When `reporting.R` is executed, the magicc reporting parts fail with
```
[1] "Executing reporting.R"
/p/tmp/aloisdir/remind-trunk/output/SDP-calibrate_2020-05-27_16.57.02/magicc
At line 1299 of file MAGICC_6005_v2_forJessicaGunnarElmar.F90 (unit = 42, file = 'MAGCFG_NMLYEARS.CFG')
Fortran runtime error: End of file
sh: climate_reporting_template.txt: No such file or directory
sed: can't read ../../../output/SDP-calibrate_2020-05-27_16.57.02/REMIND_climate_SDP-calibrate.mif: No such file or directory
cat: ../../../output/SDP-calibrate_2020-05-27_16.57.02/REMIND_climate_SDP-calibrate.mif: No such file or directory
```
Consequentially the EDGE Transport reporting is not executed (it comes after the magicc reporting in `reporting.R`) but a MIF file is produced so without checking the logs or looking for EDGE-T or MAGICC reporting, no error is evident.https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/173Subsequent REMIND runs start even if basis-run does not converge2021-02-10T14:11:09+01:00Jerome HilaireSubsequent REMIND runs start even if basis-run does not converge*Created by: giannou*
We might want to change this*Created by: giannou*
We might want to change thishttps://gitlab.pik-potsdam.de/hilaire/remind/-/issues/170full.lst lagging considerably behind full.log2020-05-12T14:35:26+02:00Jerome Hilairefull.lst lagging considerably behind full.log*Created by: giannou*
Why is it that full.log reports being at the 12th iteration whereas full.lst hasn't even started reporting the first one? Does anyone have an idea?*Created by: giannou*
Why is it that full.log reports being at the 12th iteration whereas full.lst hasn't even started reporting the first one? Does anyone have an idea?Michaja PehlMichaja Pehlhttps://gitlab.pik-potsdam.de/hilaire/remind/-/issues/174Subsequent runs not starting 2020-06-08T11:00:11+02:00Jerome HilaireSubsequent runs not starting *Created by: giannou*
This run:
`/p/tmp/giannou/sa2_updated/output/NDC_spvL_CDR_2020-05-12_20.12.24`
has 3 subsequent runs:
`$subsequentruns
[1] "PkBudg1100_spvL_CDR" "PkBudg900_spvL_CDR" "PkBudg1300_spvL_CDR"`
They weren't start...*Created by: giannou*
This run:
`/p/tmp/giannou/sa2_updated/output/NDC_spvL_CDR_2020-05-12_20.12.24`
has 3 subsequent runs:
`$subsequentruns
[1] "PkBudg1100_spvL_CDR" "PkBudg900_spvL_CDR" "PkBudg1300_spvL_CDR"`
They weren't started when the run was finished and they don't start when I restart the basis-run with
`Rscript start.R --restart` from the mail folder.
Any idea what is going on here? I get no error message and in the `log.txt` file it looks as though the subsequent runs don't exist.
Other runs of mine had the same problem.https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/172dos2unix not directly available on cluster nodes2020-07-23T10:21:03+02:00Jerome Hilairedos2unix not directly available on cluster nodes*Created by: johanneskoch94*
Correct me if I'm wrong but, as far as I can tell, the following lines in the prepare function of our starting procedure don't work
```
# Make sure all MAGICC files have LF line endings, so Fortran won't...*Created by: johanneskoch94*
Correct me if I'm wrong but, as far as I can tell, the following lines in the prepare function of our starting procedure don't work
```
# Make sure all MAGICC files have LF line endings, so Fortran won't crash
if (on_cluster)
system("find ./core/magicc/ -type f | xargs dos2unix -q")
```
They produce the following error in the log.txt file:
> xargs: dos2unix: No such file or directory
The problem is that the system on the cluster nodes can't find the dos2unix command. We need to add the dos2unix executable to our $PATH (e.g. in /p/projects/rd3mod/tools), or remove the command. (Since it isn't working, as far as I can tell, and nobody is complaining, maybe it isn't necessary?)
https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/171tax=off not compatible with cm_emiscen=12020-05-12T18:31:47+02:00Jerome Hilairetax=off not compatible with cm_emiscen=1*Created by: johanneskoch94*
Compilation error: pm_taxCO2eqHist not defined
Todo: either mark as incompatible, or add definition to the tax=off module*Created by: johanneskoch94*
Compilation error: pm_taxCO2eqHist not defined
Todo: either mark as incompatible, or add definition to the tax=off modulehttps://gitlab.pik-potsdam.de/hilaire/remind/-/issues/165Current default is non-optimal2020-05-07T16:00:48+02:00Jerome HilaireCurrent default is non-optimal*Created by: johanneskoch94*
The current default version is non-optimal. Is there some work-in-progress going on?*Created by: johanneskoch94*
The current default version is non-optimal. Is there some work-in-progress going on?https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/146SSP2-PkBudg1100 runs with latest REMIND release crash with division by zero2020-05-13T17:18:30+02:00Jerome HilaireSSP2-PkBudg1100 runs with latest REMIND release crash with division by zero*Created by: giannou*
/home/giannou/REMIND/output/SSP2-PkBudg1100_2020-04-13_16.10.08/full.lst*Created by: giannou*
/home/giannou/REMIND/output/SSP2-PkBudg1100_2020-04-13_16.10.08/full.lst