remind issueshttps://gitlab.pik-potsdam.de/hilaire/remind/-/issues2021-02-10T15:47:22+01:00https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/299Download and distribute input data error2021-02-10T15:47:22+01:00Jerome HilaireDownload and distribute input data error*Created by: fei-cheng*
Hi,
I downloaded the latest release and tried to run this model. It threw an error while trying to download and distribute input data. The error message looks like:
```
Load data..
Following files not found...*Created by: fei-cheng*
Hi,
I downloaded the latest release and tried to run this model. It threw an error while trying to download and distribute input data. The error message looks like:
```
Load data..
Following files not found:
rev5.961_690d3718e151be1b450b394c1064b1c5_remind.tgz
```
My understanding is it tries to download the above tgz file from somewhere. But where is the file stored?
And according to the README, the input data is not part of this project. But do we provide any example data for me to understand the structure and the format of the input data?
Many thanks!
https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/302reduce github repository size2021-02-10T14:06:05+01:00Jerome Hilairereduce github repository size*Created by: giannou*
*Created by: giannou*
https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/252dependency problem on Mac: regionscode function not found2020-10-06T13:49:27+02:00Jerome Hilairedependency problem on Mac: regionscode function not found*Created by: jeh0753*
Installing the latest R version, GAMS version, and following the documentation through on Mac brings the following error when running `source('start.R')`:
<img width="567" alt="image" src="https://user-images.g...*Created by: jeh0753*
Installing the latest R version, GAMS version, and following the documentation through on Mac brings the following error when running `source('start.R')`:
<img width="567" alt="image" src="https://user-images.githubusercontent.com/1597013/94822730-fac18700-03d0-11eb-8679-696b6fb23b20.png">
I've recreated this on two separate Mac machines. If I'm able to sort it out I'll post here, but any ideas or solutions would be a great help.https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/239Document values in generisdata2020-09-16T12:59:37+02:00Jerome HilaireDocument values in generisdata*Created by: Loisel*
There should be a facility to document values in generisdata, as REMIND results are quite sensitive to these assumptions. One possibility would be to start from a YAML file and generate the `.prn`, thereby ensuring ...*Created by: Loisel*
There should be a facility to document values in generisdata, as REMIND results are quite sensitive to these assumptions. One possibility would be to start from a YAML file and generate the `.prn`, thereby ensuring the presence of some documentation string (ideally containing a reference).https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/223filter out files in run status (rs), keep only directories2020-08-26T14:56:02+02:00Jerome Hilairefilter out files in run status (rs), keep only directories*Created by: giannou*
*Created by: giannou*
https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/222run status (rs) script reports wrong results in debug mode2021-02-10T14:10:46+01:00Jerome Hilairerun status (rs) script reports wrong results in debug mode*Created by: giannou*
*Created by: giannou*
https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/180Reporting error: Fortran runtime Error in the Magicc Reporting2020-10-26T13:41:21+01:00Jerome HilaireReporting error: Fortran runtime Error in the Magicc Reporting*Created by: Loisel*
When `reporting.R` is executed, the magicc reporting parts fail with
```
[1] "Executing reporting.R" ...*Created by: Loisel*
When `reporting.R` is executed, the magicc reporting parts fail with
```
[1] "Executing reporting.R"
/p/tmp/aloisdir/remind-trunk/output/SDP-calibrate_2020-05-27_16.57.02/magicc
At line 1299 of file MAGICC_6005_v2_forJessicaGunnarElmar.F90 (unit = 42, file = 'MAGCFG_NMLYEARS.CFG')
Fortran runtime error: End of file
sh: climate_reporting_template.txt: No such file or directory
sed: can't read ../../../output/SDP-calibrate_2020-05-27_16.57.02/REMIND_climate_SDP-calibrate.mif: No such file or directory
cat: ../../../output/SDP-calibrate_2020-05-27_16.57.02/REMIND_climate_SDP-calibrate.mif: No such file or directory
```
Consequentially the EDGE Transport reporting is not executed (it comes after the magicc reporting in `reporting.R`) but a MIF file is produced so without checking the logs or looking for EDGE-T or MAGICC reporting, no error is evident.https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/173Subsequent REMIND runs start even if basis-run does not converge2021-02-10T14:11:09+01:00Jerome HilaireSubsequent REMIND runs start even if basis-run does not converge*Created by: giannou*
We might want to change this*Created by: giannou*
We might want to change thishttps://gitlab.pik-potsdam.de/hilaire/remind/-/issues/174Subsequent runs not starting 2020-06-08T11:00:11+02:00Jerome HilaireSubsequent runs not starting *Created by: giannou*
This run:
`/p/tmp/giannou/sa2_updated/output/NDC_spvL_CDR_2020-05-12_20.12.24`
has 3 subsequent runs:
`$subsequentruns
[1] "PkBudg1100_spvL_CDR" "PkBudg900_spvL_CDR" "PkBudg1300_spvL_CDR"`
They weren't start...*Created by: giannou*
This run:
`/p/tmp/giannou/sa2_updated/output/NDC_spvL_CDR_2020-05-12_20.12.24`
has 3 subsequent runs:
`$subsequentruns
[1] "PkBudg1100_spvL_CDR" "PkBudg900_spvL_CDR" "PkBudg1300_spvL_CDR"`
They weren't started when the run was finished and they don't start when I restart the basis-run with
`Rscript start.R --restart` from the mail folder.
Any idea what is going on here? I get no error message and in the `log.txt` file it looks as though the subsequent runs don't exist.
Other runs of mine had the same problem.https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/172dos2unix not directly available on cluster nodes2020-07-23T10:21:03+02:00Jerome Hilairedos2unix not directly available on cluster nodes*Created by: johanneskoch94*
Correct me if I'm wrong but, as far as I can tell, the following lines in the prepare function of our starting procedure don't work
```
# Make sure all MAGICC files have LF line endings, so Fortran won't...*Created by: johanneskoch94*
Correct me if I'm wrong but, as far as I can tell, the following lines in the prepare function of our starting procedure don't work
```
# Make sure all MAGICC files have LF line endings, so Fortran won't crash
if (on_cluster)
system("find ./core/magicc/ -type f | xargs dos2unix -q")
```
They produce the following error in the log.txt file:
> xargs: dos2unix: No such file or directory
The problem is that the system on the cluster nodes can't find the dos2unix command. We need to add the dos2unix executable to our $PATH (e.g. in /p/projects/rd3mod/tools), or remove the command. (Since it isn't working, as far as I can tell, and nobody is complaining, maybe it isn't necessary?)
https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/171tax=off not compatible with cm_emiscen=12020-05-12T18:31:47+02:00Jerome Hilairetax=off not compatible with cm_emiscen=1*Created by: johanneskoch94*
Compilation error: pm_taxCO2eqHist not defined
Todo: either mark as incompatible, or add definition to the tax=off module*Created by: johanneskoch94*
Compilation error: pm_taxCO2eqHist not defined
Todo: either mark as incompatible, or add definition to the tax=off modulehttps://gitlab.pik-potsdam.de/hilaire/remind/-/issues/145Single-region runs started via config file report OUT_OF_MEMORY2020-05-08T16:00:34+02:00Jerome HilaireSingle-region runs started via config file report OUT_OF_MEMORY*Created by: Loisel*
When starting a `testOneRegi` run on the cluster by changing the config file instead of using the `--testOneRegi` flag to `start.R`, the run reports via email (subject, body is empty)
```
SLURM Job_id=16067591 Nam...*Created by: Loisel*
When starting a `testOneRegi` run on the cluster by changing the config file instead of using the `--testOneRegi` flag to `start.R`, the run reports via email (subject, body is empty)
```
SLURM Job_id=16067591 Name=default Ended, Run time 00:09:58, OUT_OF_MEMORY
```
Checking the output, `/p/tmp/aloisdir/remind/output/default_2020-04-16_10.30.13`, everything seems to be fine (except MAGICC) but validation seems to fail. Log says
```
[1] "Executing reporting.R"
/p/tmp/aloisdir/remind/output/default_2020-04-16_10.30.13/magicc
At line 1299 of file MAGICC_6005_v2_forJessicaGunnarElmar.F90 (unit = 42, file = 'MAGCFG_NMLYEARS.CFG')
Fortran runtime error: End of file
sh: climate_reporting_template.txt: No such file or directory
sed: can't read ../../../output/default_2020-04-16_10.30.13/REMIND_climate_default.mif: No such file or directory
cat: ../../../output/default_2020-04-16_10.30.13/REMIND_climate_default.mif: No such file or directory
Joining, by = c("region", "tech", "rlf")
Joining, by = "opTimeYr"
Joining, by = "tech"
Joining, by = c("opTimeYr", "fuel")
Joining, by = c("region", "tech")
Joining, by = c("region", "tech")
Joining, by = c("region", "period", "tech")
Joining, by = "tech"
Joining, by = c("region", "period", "tech")
Joining, by = c("region", "period", "tech")
[1] "Executing validation.R"
/var/spool/slurmd/job16067591/slurm_script: line 4: 21304 Killed Rscript prepare_and_run.R
slurmstepd: error: Detected 1 oom-kill event(s) in step 16067591.batch cgroup.
```
REMIND version is `998771da3af9b243925161105ff86e5cd865b0d9`, no changes to config apart from `cfg$gms$optimization <- "testOneRegi" `.
https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/144Calibration of USA fesob fails2020-04-24T17:41:50+02:00Jerome HilaireCalibration of USA fesob fails*Created by: piklev*
The calibration of fesob in the USA region does not reach the target, even after many iterations:
SDP:
![Screenshot from 2020-04-09 19-20-20](https://user-images.githubusercontent.com/6737625/79339960-f5a77300-7...*Created by: piklev*
The calibration of fesob in the USA region does not reach the target, even after many iterations:
SDP:
![Screenshot from 2020-04-09 19-20-20](https://user-images.githubusercontent.com/6737625/79339960-f5a77300-7f29-11ea-9652-821f3384fec9.png)
SSP2:
![Screenshot from 2020-04-09 19-33-22](https://user-images.githubusercontent.com/6737625/79339965-f6d8a000-7f29-11ea-9fe4-4ef1b9a102d8.png)
Behind the quantities, the prices of fesob show great variations, with sudden price drops or increases from one period to another.
In the calibration routine, the strong and sudden variations of prices are smoothed. This normally does not cause a problem, but if prices vary strongly, the difference between smoothed and original prices becomes important. Therefore the computed efficiencies (which make target quantities optimal at the smoothed prices) will not adapt so as to make the output quantities close to the target quantities (because smoothed prices are too far away from output prices). Removing the smoothing would help for fesos USA, but it would cause problems elsewhere. The real problem lies in the strong and sudden variations of prices, which I could not solve.
Solids, and especially solids in buildings, are one of the most problematic energy carriers in the calibration, because there are many equations and fixings which constrain their development, and prices. I list some of the constraints and equations I played around with in the following. Unfortunately, this has not helped solving the price issue for fesob/i USA so far. Prices of fesob and fesoi should normally be equal. There are several reasons why it need not be the case.
- FE taxes and subsidies: normally, taxes and subsidies are linearly reduced until 2050 when they reach 0. So before 2050, taxes and subsidies can justify a difference in prices between fesob and fesoi
- `q_limitBiotrmod`: this equation is in [`core/equations.gms`](https://github.com/remindmodel/remind/blob/998771da3af9b243925161105ff86e5cd865b0d9/core/equations.gms#L471) . It is supposed to prevent that all solids are provided by biomass in the long term, at least for industry. In practice, this equation says that `(modern biomass in B and I)-(solids in B)<2*(coal in B and I)`. The equation is less stringent in the short term (5 instead of 2). This equation, if at the boundary, can lead to a difference in prices of fesob and fesoi.
- `q_inconvPenCoalSolids`, now `q02_inconvPenCoalSolids`, in [`modules/02_welfare/utilitarian/equations.com`](https://github.com/remindmodel/remind/blob/998771da3af9b243925161105ff86e5cd865b0d9/modules/02_welfare/utilitarian/equations.gms#L53). Local air pollution for coal. But in practice, the penalty applies to (coal for B and I) – (solids in I). So this equation can also lead to a difference in prices of fesob and fesoi.
- [`vm_deltaCap.fx` (biotr)](https://github.com/remindmodel/remind/blob/998771da3af9b243925161105ff86e5cd865b0d9/core/bounds.gms#L58): in most scenarios biotr follows a fixed trajectory. But it cannot be suspected to cause variations in fesob/i prices as biotr is 0 in the USA.
In addition, I have tried to fix fesob and fesoi to their 2025 target quantity: the idea is to provoke an infeasibility that could point to the original problem (for early periods it is usually related to capacities). There was no infeasibility.
Instead of looking directly for the source of the problem, another option to solve the calibration issue could be track back from which version on the problem occurs. This would require making a calibration of several versions of the model. But this option is a rather expensive way of solving the issue, and it could probably only work if the problem arises from a significant change and not from an accumulation of small changes.
https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/133Blocking 12+ CPU's for run setup2020-05-11T14:44:59+02:00Jerome HilaireBlocking 12+ CPU's for run setup*Created by: giannou*
..is a waste of resources. Maybe a two-step process is needed, like the one we had before?*Created by: giannou*
..is a waste of resources. Maybe a two-step process is needed, like the one we had before?https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/126Running magicc separately in REMIND output folder does not work2020-03-27T11:06:54+01:00Jerome HilaireRunning magicc separately in REMIND output folder does not work*Created by: merfort*
*Created by: merfort*
https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/76missing R dependencies for installing REMIND2020-02-21T15:01:54+01:00Michaja Pehlmissing R dependencies for installing REMINDhttps://github.com/remindmodel/remind/blob/23fc5eb1677378dc2247839c886ac1417eab9c71/tutorials/1_GettingREMIND.md#L30-L45
This fails, because `remind` depends on `remulator`, depends on `magpie4`, depends on `moinput`, depends on `rhdf5`...https://github.com/remindmodel/remind/blob/23fc5eb1677378dc2247839c886ac1417eab9c71/tutorials/1_GettingREMIND.md#L30-L45
This fails, because `remind` depends on `remulator`, depends on `magpie4`, depends on `moinput`, depends on `rhdf5`. So the Bioconductor-stuff is needed as well.https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/62question on path_gdx_bau2020-02-11T15:44:21+01:00Michaja Pehlquestion on path_gdx_bau`path_gdx_bau` is not included in the default configuration
https://github.com/remindmodel/remind/blob/46de32775ed14828d0a095a80c18e508d3cce8d5/config/scenario_config.csv#L1
which gives a bothersome warning in `start.R`
https://github...`path_gdx_bau` is not included in the default configuration
https://github.com/remindmodel/remind/blob/46de32775ed14828d0a095a80c18e508d3cce8d5/config/scenario_config.csv#L1
which gives a bothersome warning in `start.R`
https://github.com/remindmodel/remind/blob/46de32775ed14828d0a095a80c18e508d3cce8d5/start.R#L84
```
Warning messages:
1: In is.na(isettings$path_gdx_bau) :
is.na() applied to non-(list or vector) of type 'NULL'
```
Shouldn't it be included in the default configuration (even if empty)?https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/52Is there a legitimate reason for everybody and his brother being part of `enty`?2020-11-18T09:22:33+01:00Michaja PehlIs there a legitimate reason for everybody and his brother being part of `enty`?`enty` used to be "energy type" and encompass primary, secondary, and final energy (`pety`, `sety`, `fety`). Now it includes useful energy, a gazillion emissions, `peog` as an oil-and gas aggregate, as well as the `good` and `perm` trad...`enty` used to be "energy type" and encompass primary, secondary, and final energy (`pety`, `sety`, `fety`). Now it includes useful energy, a gazillion emissions, `peog` as an oil-and gas aggregate, as well as the `good` and `perm` tradable assets.
Is there a legitimate case where they all have to be in the same set? Because that's blowing up _all_ the variables and parameters involved and is quite confusing from time to time.Michaja PehlMichaja Pehlhttps://gitlab.pik-potsdam.de/hilaire/remind/-/issues/41Weired special case in q_emiTeDetail2020-02-11T16:00:01+01:00Michaja PehlWeired special case in q_emiTeDetailhttps://github.com/remindmodel/remind/blob/08e58efe626352b7963575d3c6e13436e3322dd4/core/equations.gms#L478-L502
I don't see any point in (and am confused by) the `OR (pe2se(enty,enty2,te) AND sameas(enty3,"cco2"))` part. This would ...https://github.com/remindmodel/remind/blob/08e58efe626352b7963575d3c6e13436e3322dd4/core/equations.gms#L478-L502
I don't see any point in (and am confused by) the `OR (pe2se(enty,enty2,te) AND sameas(enty3,"cco2"))` part. This would come into play for combinations of `(pety,sety,te,"cco2")` that are not present in `emi2te` yet have non-zero data in `pm_emifac`. This might happen if new data is included in `./core/input/generisdata_emi.put`, but is not the case at the moment.
However, these entries would be excluded from the equation in any case, since it sums over `emi2te`, of which they are not a part. (Incidentally, this sum does nothing productive besides excluding the non-existent cases that were manually included.)
Do you have any idea what was the reasoning behind this? I will delete it, unless there's some foreseeable edge-case where this is necessary.https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/7c_GDPpcScen flag crashes code2019-12-13T12:13:44+01:00Jerome Hilairec_GDPpcScen flag crashes code*Created by: giannou*
![Screenshot 2019-12-10 16 17 18](https://user-images.githubusercontent.com/11047746/70542229-aced4100-1b68-11ea-8b68-4741a3f19c4b.png)
*Created by: giannou*
![Screenshot 2019-12-10 16 17 18](https://user-images.githubusercontent.com/11047746/70542229-aced4100-1b68-11ea-8b68-4741a3f19c4b.png)
https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/38rs (run statistics) does not work2020-01-20T11:58:20+01:00Jerome Hilairers (run statistics) does not work*Created by: giannou*
due to the renaming of log.txt to $JOBNAME.out in https://github.com/remindmodel/remind/commit/285f5c23dc3d79c4e89dd510778d0f20c93b1343*Created by: giannou*
due to the renaming of log.txt to $JOBNAME.out in https://github.com/remindmodel/remind/commit/285f5c23dc3d79c4e89dd510778d0f20c93b1343https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/54add explicit reporting of biomass in industry FE use2020-02-04T10:40:53+01:00Michaja Pehladd explicit reporting of biomass in industry FE useMichaja PehlMichaja Pehlhttps://gitlab.pik-potsdam.de/hilaire/remind/-/issues/170full.lst lagging considerably behind full.log2020-05-12T14:35:26+02:00Jerome Hilairefull.lst lagging considerably behind full.log*Created by: giannou*
Why is it that full.log reports being at the 12th iteration whereas full.lst hasn't even started reporting the first one? Does anyone have an idea?*Created by: giannou*
Why is it that full.log reports being at the 12th iteration whereas full.lst hasn't even started reporting the first one? Does anyone have an idea?Michaja PehlMichaja Pehlhttps://gitlab.pik-potsdam.de/hilaire/remind/-/issues/113start.R --restart does not submit slurm jobs, runs on login node instead2020-04-30T12:03:43+02:00Michaja Pehlstart.R --restart does not submit slurm jobs, runs on login node insteadhttps://gitlab.pik-potsdam.de/hilaire/remind/-/issues/129git version and changes written somewhere in the run?2020-04-23T17:55:48+02:00Jerome Hilairegit version and changes written somewhere in the run?*Created by: giannou*
We used to have the SVN/git version and the changes with respect to these reported in the run files, is this not the case any more? It was a very helpful feature.*Created by: giannou*
We used to have the SVN/git version and the changes with respect to these reported in the run files, is this not the case any more? It was a very helpful feature.https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/134current version is broken2020-04-07T13:12:00+02:00Jerome Hilairecurrent version is broken*Created by: giannou*
out-of-the-box REMIND run with module piam/1.10 loaded fails with error message:
Error in setNames(pop[regi, , ] * log(1000 * cons[regi, , ] * (1 - (c_damage * :
dims [product 19] do not match the length of ...*Created by: giannou*
out-of-the-box REMIND run with module piam/1.10 loaded fails with error message:
Error in setNames(pop[regi, , ] * log(1000 * cons[regi, , ] * (1 - (c_damage * :
dims [product 19] do not match the length of object [0]
Calls: mbind -> reportMacroEconomy -> setNames
after the 1st iteration, when running exoGAINSAirpollutants.Rhttps://gitlab.pik-potsdam.de/hilaire/remind/-/issues/55rename industry UE reporting2020-02-04T10:40:31+01:00Michaja Pehlrename industry UE reportingMichaja PehlMichaja Pehlhttps://gitlab.pik-potsdam.de/hilaire/remind/-/issues/56document calculation of costs from MAC curves2020-02-11T15:59:13+01:00Michaja Pehldocument calculation of costs from MAC curvesMichaja PehlMichaja Pehlhttps://gitlab.pik-potsdam.de/hilaire/remind/-/issues/274REMIND Run Portability Snags2021-01-04T14:21:30+01:00Michaja PehlREMIND Run Portability SnagsThe idea used to be that a REMIND run can be used from its directory independent of all other REMIND code. Currently, that's not the case any more in at least two instances. I copied a run to a temporary directory for debugging and got...The idea used to be that a REMIND run can be used from its directory independent of all other REMIND code. Currently, that's not the case any more in at least two instances. I copied a run to a temporary directory for debugging and got this:
```
$ grep Error log.txt | sort | uniq
Error in fread("../../modules/35_transport/edge_esm/input/EDGEscenario_description.csv") :
Error in read.magpie("../../modules/11_aerosols/exoGAINS/input/ef_gains.cs4r")
```
So at least two files need to be included in https://github.com/remindmodel/remind/blob/be7a866d4318ce78cd09a92b7f9637421dd3f106/config/default.cfg#L460-L475 and scripts need to be adjusted.https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/286geohdr infeasibilities2020-12-03T17:05:55+01:00Michaja Pehlgeohdr infeasibilities**tl;dr:** EDGE-Transport needs to stop messing with the past like a naughty time traveller. ;)
I'm pretty confident I found the origin of the infeasibilities having to do with `geohdr` capacity in the first free time step.
Ultimat...**tl;dr:** EDGE-Transport needs to stop messing with the past like a naughty time traveller. ;)
I'm pretty confident I found the origin of the infeasibilities having to do with `geohdr` capacity in the first free time step.
Ultimately, it is caused by `EDGE_transport.R` modifying the values of `p35_fe2es`
https://github.com/remindmodel/remind/blob/547ba8f7be1bfc8adbd9cf1c39438089e9c0b174/modules/35_transport/edge_esm/presolve.gms#L13-L19
These end up in the `fulldata.gdx`, which becomes the `input_ref.gdx` for the next run, where they are loaded, overwriting the data from `./modules/35_transport/edge_esm/input/fe2es.cs4r`.
https://github.com/remindmodel/remind/blob/547ba8f7be1bfc8adbd9cf1c39438089e9c0b174/modules/35_transport/edge_esm/datainput.gms#L50-L54
Then, different `pm_fe2es` values change the result of `initialcap2`, specifically `v05_INIcap0`/`pm_cap0`, which trickles down to change the vintage capital structure of `geohdr` through `pm_cap0` → `s05_aux_tot_prod` → `s05_aux_prod_remaining` → `p05_aux_prod_thisgrade` → `p05_aux_cap_distr` → `p05_aux_cap` → `sm_tmp` → `vm_capDistr.fx`:
https://github.com/remindmodel/remind/blob/547ba8f7be1bfc8adbd9cf1c39438089e9c0b174/modules/05_initialCap/on/preloop.gms#L145-L182
Then, depending on region and scenario settings, it is possible that (a) CONOPT is in a weird spot it can't find out of (my Npi run) or (b) the modified vintage capacity is so high that _capacity × capacity factor_ exceeds `pm_dataren(regi,"maxprod",rlf,te)` (Felix` NSW run) – which isn't a problem for the fixed time-steps, as `q_limitProd` is [defined only over the free ones](https://github.com/remindmodel/remind/blob/547ba8f7be1bfc8adbd9cf1c39438089e9c0b174/core/equations.gms#L400-L407).
@LaviniaBaumstark @dklein-pik @giannou (In lieu of a RSE github team):
I think it's problematic that a run, that is supposed to be fixed on a different run up to some point, has a different vintage structure. I was always puzzled by the fact that `initialcap2` is run for _every run_. This seems not only to be the case for `geohdr`, but other technologies too. Small thing that doesn't show up in the results (`vm_cap` is still the same), but this bug wouldn't happened without `initialcap2` running again, although its results should be the same in any case.
In any case, preserving the 2005 values of `pm_fe2es` at the `./modules/35_transport/edge_esm/input/fe2es.cs4r` levels should fix this.
-----
Evidence/notes using the failed DeepEl3 Npi run as an example: `/p/tmp/pehl/Remind_DeepEl/output/TraInd-Npi_2020-11-27_16.06.51`
```
$ gdxdiff input_ref.gdx non_optimal.gdx id=pm_cap0,pm_EN_demand_from_initialcap2,pm_cesdata,pm_shFeCes,pm_fe2es,pm_cf,pm_data,pm_prodCouple,p35_fe2es,EDGE_scenario
GDXDIFF 31.1.1 r4b06116 Released May 16, 2020 LEG x86 64bit/Linux
File1 : input_ref.gdx
File2 : non_optimal.gdx
Id : pm_cap0 pm_EN_demand_from_initialcap2 pm_cesdata pm_shFeCes pm_fe2es pm_cf pm_data pm_prodCouple p35_fe2es EDGE_scenario
Summary of differences:
pm_cap0 Data are different
pm_cf Data are different
pm_data Data are different
pm_EN_demand_from_initialcap2 Data are different
pm_fe2es Data are different
pm_shFeCes Data are different
Output: diffile.gdx
GDXDiff finished
```
```
$ dumpgdx diffile.gdx vm_deltaCap "CHA.*geohdr.*L\>" | head
1995 CHA geohdr 1 dif1 L 8.2494690248504E-8
1995 CHA geohdr 1 dif2 L 8.24965476551363E-8
2000 CHA geohdr 1 dif1 L 1.42715686791135E-6
2000 CHA geohdr 1 dif2 L 1.42718900101742E-6
2015 CHA geohdr 1 dif1 L 0.000906719440325381
2015 CHA geohdr 1 dif2 L 0.000906719440325381
2020 CHA geohdr 1 dif1 L 3.4181901481737E-5
2020 CHA geohdr 1 dif2 L 3.4181901481737E-5
2030 CHA geohdr 1 dif1 L 0.000100553100549173
2030 CHA geohdr 1 dif2 L 0.000100553067898506
```
```
$ dumpgdx diffile.gdx pm_cap0 "CHA.*geohdr"
CHA geohdr dif1 1.49901612654113E-5
CHA geohdr dif2 1.4990498775921E-5
```
```
$ dumpgdx diffile.gdx pm_fe2es "2005.*CHA"
2005 CHA te_espet_pass_sm dif1 32.7810653744073
2005 CHA te_espet_pass_sm dif2 32.7803397536317
2005 CHA te_esdie_pass_sm dif1 72.8208842393301
2005 CHA te_esdie_pass_sm dif2 72.6703864713621
2005 CHA te_eselt_pass_sm dif1 274.077206442013
2005 CHA te_eselt_pass_sm dif2 276.29708722215
2005 CHA te_esh2t_pass_sm dif1 64.7583317381968
2005 CHA te_esh2t_pass_sm dif2 64.742052256529
2005 CHA te_esgat_pass_sm dif1 66.8262951982073
2005 CHA te_esgat_pass_sm dif2 67.1300018776568
2005 CHA te_esdie_frgt_sm dif1 35.041588297353
2005 CHA te_esdie_frgt_sm dif2 34.9779610388119
2005 CHA te_eselt_frgt_sm dif1 226.082090011469
2005 CHA te_eselt_frgt_sm dif2 226.082090011469
2005 CHA te_esh2t_frgt_sm dif1 33.3906087857954
2005 CHA te_esh2t_frgt_sm dif2 33.391690768669
2005 CHA te_esgat_frgt_sm dif1 13.2919587450842
2005 CHA te_esgat_frgt_sm dif2 13.3447480682688
```https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/165Current default is non-optimal2020-05-07T16:00:48+02:00Jerome HilaireCurrent default is non-optimal*Created by: johanneskoch94*
The current default version is non-optimal. Is there some work-in-progress going on?*Created by: johanneskoch94*
The current default version is non-optimal. Is there some work-in-progress going on?https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/86asked for Negishi, scripts start Nash2020-03-02T15:00:37+01:00Jerome Hilaireasked for Negishi, scripts start Nash*Created by: giannou*
![Screenshot 2020-03-02 12 41 33](https://user-images.githubusercontent.com/11047746/75673593-9f38ea80-5c83-11ea-9407-1b4608b6e92b.png)
*Created by: giannou*
![Screenshot 2020-03-02 12 41 33](https://user-images.githubusercontent.com/11047746/75673593-9f38ea80-5c83-11ea-9407-1b4608b6e92b.png)
https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/117Usage of start.R is not documented2020-04-14T09:47:31+02:00Jerome HilaireUsage of start.R is not documented*Created by: Loisel*
Please indicate at the top of the file how it is used and which parameters or flags can be given.
Alternatively, one could use `optparse` and provide `--help` or `--usage`.*Created by: Loisel*
Please indicate at the top of the file how it is used and which parameters or flags can be given.
Alternatively, one could use `optparse` and provide `--help` or `--usage`.https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/139How can the SLURM parameter '#SBATCH --time' be adjusted?2020-04-23T13:25:04+02:00Jerome HilaireHow can the SLURM parameter '#SBATCH --time' be adjusted?*Created by: giannou*
*Created by: giannou*
https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/112start.R fails to restart fixed runs2020-04-07T16:24:56+02:00Michaja Pehlstart.R fails to restart fixed runsby not decompressing `levs.gms.gz`, `margs.gms.gz`, and `fixings.gms.gz`.by not decompressing `levs.gms.gz`, `margs.gms.gz`, and `fixings.gms.gz`.https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/140How can a run be restarted, that was interrupted before a fulldata.gdx has be...2020-04-22T17:05:37+02:00Jerome HilaireHow can a run be restarted, that was interrupted before a fulldata.gdx has been created?*Created by: giannou*
*Created by: giannou*
https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/181How is the LCOEplot reporting supposed to be used?2020-06-01T14:18:51+02:00Jerome HilaireHow is the LCOEplot reporting supposed to be used?*Created by: giannou*
https://github.com/remindmodel/remind/blob/develop/scripts/output/single/LCOEPlot.R
throws an error when called via "Rscript output.R"
*Created by: giannou*
https://github.com/remindmodel/remind/blob/develop/scripts/output/single/LCOEPlot.R
throws an error when called via "Rscript output.R"
https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/193Short-term issues2020-06-19T12:26:22+02:00Jerome HilaireShort-term issues*Created by: amnmalik*
For Baseline scenarios -
- Near-term trend for SE|Gases|Coal (GLO) doesn't match to bottom-up data.
- Near-term trend for SE|Solids|Coal (GLO) doesn't match to bottom-up data.
- For SE|Solids|Coal, historical v...*Created by: amnmalik*
For Baseline scenarios -
- Near-term trend for SE|Gases|Coal (GLO) doesn't match to bottom-up data.
- Near-term trend for SE|Solids|Coal (GLO) doesn't match to bottom-up data.
- For SE|Solids|Coal, historical values not matched for JPN, REF.
- For SE|Solids|Coal, near-term trends don't match for CHA.
For SDP scenarios -
- PE|Coal - GLO, CHA, NEU, SSA doesn't match short-term trends.
- Emi|CO2|Energy|Supply|Electricity for LAM doesn't match recent trends.
- Emi|CO2|Energy|Supply|Electricity, historical mismatch for EUR
- Emi|CO2|Gross Fossil Fuels and Industry, recent trends not matched for NEU
- calibration USA feso -A spike in demand around 2020 (FE|Buildings|Solids)
General -
- SE Gases from biomass not in line with historical data (CAZ, CHA, EUR, LAM, MEA, OAS)
- SE Gases from coal totally off in CHA, IND, REF, SSA, OAS
- Less PE Gas in CAZ and REF compared with historical data (2010-2015)
- SSP2 - NDC - Consumption losses different across scenarios before 2020
- SE Electricity from gas not in line with historical data in CAZ (also IND, NEU, REF to a much smaller extent)
- High policy costs for CAZ under SSP2 NDC
https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/146SSP2-PkBudg1100 runs with latest REMIND release crash with division by zero2020-05-13T17:18:30+02:00Jerome HilaireSSP2-PkBudg1100 runs with latest REMIND release crash with division by zero*Created by: giannou*
/home/giannou/REMIND/output/SSP2-PkBudg1100_2020-04-13_16.10.08/full.lst*Created by: giannou*
/home/giannou/REMIND/output/SSP2-PkBudg1100_2020-04-13_16.10.08/full.lsthttps://gitlab.pik-potsdam.de/hilaire/remind/-/issues/53v_co2capturevalve is venting industry CCS2020-02-05T11:44:28+01:00Michaja Pehlv_co2capturevalve is venting industry CCShttps://github.com/remindmodel/remind/blob/c67cc61e1ef4d6a36c7dd487bc58e22f51294ae8/core/equations.gms#L696-L702
c.f. `default_2020-01-08_14.13.34` run, 2020/USA
```
vm_co2capture 2020 USA cco2 ico2 ccsinje 1 L 0.0030434...https://github.com/remindmodel/remind/blob/c67cc61e1ef4d6a36c7dd487bc58e22f51294ae8/core/equations.gms#L696-L702
c.f. `default_2020-01-08_14.13.34` run, 2020/USA
```
vm_co2capture 2020 USA cco2 ico2 ccsinje 1 L 0.00304342219022761
v_co2capturevalve 2020 USA L 0.00194586349440526
vm_co2CCS 2020 USA cco2 ico2 ccsinje 1 L 0.00109755869582235
vm_emiTeDetail 2020 USA * * * cco2 L 0.00036196011504294
vm_ccs_cdr 2020 USA cco2 ico2 ccsinje * L 0
vm_emiIndCCS 2020 USA * L 0.00268146207518457
```
`vm_co2capture` = `vm_co2CCS` + `v_co2capturevalve` and
`vm_co2CCS` = `vm_emiTeDetail` + `vm_ccs_cdr` + `vm_emiIndCCS`
So CO2 captured in industry gets vented. I'm not entirely sure what the purpose of `v_co2capturevalve` is
https://github.com/remindmodel/remind/blob/c67cc61e1ef4d6a36c7dd487bc58e22f51294ae8/core/declarations.gms#L293
but if industry is supposed to supply CO2 for CCU, that should be modelled explicitly.https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/265iteration/resource limits on CONOPT2020-10-27T13:04:08+01:00Michaja Pehliteration/resource limits on CONOPThttps://github.com/remindmodel/remind/blob/3dc35199bbf5544b148d2e9e2f706761220cec63/core/loop.gms#L18-L21
Could we release these limits on CONOPT?
I'm mainly interested in `reslim` for Nash, because the REMIND-EU calibration has sh...https://github.com/remindmodel/remind/blob/3dc35199bbf5544b148d2e9e2f706761220cec63/core/loop.gms#L18-L21
Could we release these limits on CONOPT?
I'm mainly interested in `reslim` for Nash, because the REMIND-EU calibration has shown to need more time to find a feasible solution (at least three and a half hours). I don't think limiting `reslim` is useful (anymore).
- Most runs solve faster, so this limit isn't reached very often (aka does not matter).
- If CONOPT does not find a solution, I would like it to keep trying. It is likely to stop at some point because the gradient gets too flat or it hits a `NA`. _But if it doesn't_, having it keep going may give me a feasible (unconverged) .gdx from which to start a new run. (Worked for me a lot for initial calibrations of new parametrisations.)
- When CONOPT stops because of a time-out, it is not likely to find a feasible solution within two hours in the next `solve` iteration or even Nash iteration, but will just waste eleven iterations trying again from the beginning, without informing the user who could then intervene.
- Runs are limited by slurm in any case, so we don't run the risk of wasting more cluster resources if this is dropped.
The non-Nash `reslim` (eleven days) isn't really different from the default (317 years) – most runs will get cancelled well before, and if not (Negishi?), users are likely to check in on them.
Same argument for `iterlim`. 1e6 is a random number that is not qualitatively different from the default (2e9).https://gitlab.pik-potsdam.de/hilaire/remind/-/issues/189make the run status (rs) script work for coupled runs also2020-09-18T11:54:10+02:00Jerome Hilairemake the run status (rs) script work for coupled runs also*Created by: giannou*
*Created by: giannou*