How to change working directory in scons - directory

I have a project built with make, but I want to shift to scons.
However, I could not link object files in scons, so I want to know how to change working directory in scons.
What I exactly want is
make -C $(OBJECTDIRECTORY) -f $(SOURCEDIRECTORY)./Makefile InternalDependency
This is one line from my Makefile, and works well.
However, when scons builds my project, it does
x86_64-pc-linux-ld -o build/kernel32/kernel32.elf -melf_i386 -T scripts/elf_i386.x -nostdlib -e main -Ttext 0x10200 build/kernel32/asmUtils.o build/kernel32/cpu.o build/kernel32/main.o build/kernel32/memory.o build/kernel32/pageManager.o build/kernel32/utils.o
and got an error,
x86_64-pc-linux-ld: cannot find main.o
. Even thouh I do same command in shell manually, I got the same error.
However, if I move to build/kernel32, and do manually
x86_64-pc-linux-ld -o kernel32.elf -melf_i386 -T ../../elf_i386.x -nostdlib -e main -Ttext 0x10200 main.o cpu.o memory.o pageManager.o utils.o asmUtils.o
and it works.
My assumption is ld could not link object files in some upper directory.
So, is there any way to do like "-C" option of Make?
Or any other workaround wayin scons?
Here is my SConscript, and SConsctruct.
In project root directory,
#SConstruct
build_dir = 'build'
# Build
SConscript(['src/SConscript'], variant_dir = build_dir, duplicate = 0)
# Clean
Clean('.', build_dir)
In src directory
#SConscript for src
SConscript(['bootloader/SConscript',
'kernel32/SConscript'])
In kernel32 directory
#SConscript for kernel32
import os, sys
# Build entry
env_entry = Environment(tools=['default', 'nasm'])
target_entry = 'entry.bin'
object_entry = 'entry.s'
output_entry = env_entry.Object(target_entry, object_entry)
# Compile CPP
env_gpp_options = {
'CXX' : 'x86_64-pc-linux-g++',
'CXXFLAGS' : '-std=c++11 -g -m32 -ffreestanding -fno-exceptions -fno-rtti',
'LINK' : 'x86_64-pc-linux-ld',
'LINKFLAGS' : '-melf_i386 -T scripts/elf_i386.x -nostdlib -e main -Ttext 0x10200',
}
env_gpp = Environment(**env_gpp_options)
env_gpp.Append(ENV = {'PATH' : os.environ['PATH']})
object_cpp_list = Glob('*.cpp')
for object_cpp in object_cpp_list:
env_gpp.Object(object_cpp)
# Compile ASM
env_nasm = Environment(tools=['default', 'nasm'])
env_nasm.Append(ASFLAGS='-f elf32')
object_nasm_list = Glob('*.asm')
for object_nasm in object_nasm_list:
env_nasm.Object(object_nasm)
# Find all object file
object_target_list = Glob('*.o')
object_target_list.append('entry.bin')
# Linking
env_link_target = 'kernel32.elf'
env_gpp.Program(env_link_target, object_target_list)
Pleas let me know. Thank you.
The log for "--tree=prune" is
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
x86_64-pc-linux-ld -o build/kernel32/kernel32.elf -melf_i386 -T scripts/elf_i386.x -nostdlib -e main -Ttext 0x10200 build/kernel32/asmUtils.o build/kernel32/cpu.o build/kernel32/main.o build/kernel32/memory.o build/kernel32/pageManager.o build/kernel32/utils.o build/kernel32/entry.bin
+-.
+-SConstruct
+-build
| +-src/SConscript
| +-build/bootloader
| | +-src/bootloader/BootLoader.asm
| | +-src/bootloader/SConscript
| | +-build/bootloader/bootloader.bin
| | +-src/bootloader/BootLoader.asm
| | +-/usr/bin/nasm
| +-build/kernel32
| +-src/kernel32/SConscript
| +-src/kernel32/asmUtils.asm
| +-build/kernel32/asmUtils.o
| | +-src/kernel32/asmUtils.asm
| | +-/usr/bin/nasm
| +-src/kernel32/cpu.cpp
| +-build/kernel32/cpu.o
| | +-src/kernel32/cpu.cpp
| | +-src/kernel32/cpu.hpp
| | +-src/kernel32/types.hpp
| | +-/home/xaliver/BuildTools/cross/bin/x86_64-pc-linux-g++
| +-build/kernel32/entry.bin
| | +-src/kernel32/entry.s
| | +-/usr/bin/nasm
| +-src/kernel32/entry.s
| +-build/kernel32/kernel32.elf
| | +-[build/kernel32/asmUtils.o]
| | +-[build/kernel32/cpu.o]
| | +-build/kernel32/main.o
| | | +-src/kernel32/main.cpp
| | | +-src/kernel32/cpu.hpp
| | | +-src/kernel32/memory.hpp
| | | +-src/kernel32/types.hpp
| | | +-src/kernel32/utils.hpp
| | | +-src/kernel32/pageManager.hpp
| | | +-src/kernel32/page.hpp
| | | +-/home/xaliver/BuildTools/cross/bin/x86_64-pc-linux-g++
| | +-build/kernel32/memory.o
| | | +-src/kernel32/memory.cpp
| | | +-src/kernel32/memory.hpp
| | | +-src/kernel32/pageManager.hpp
| | | +-src/kernel32/page.hpp
| | | +-src/kernel32/types.hpp
| | | +-/home/xaliver/BuildTools/cross/bin/x86_64-pc-linux-g++
| | +-build/kernel32/pageManager.o
| | | +-src/kernel32/pageManager.cpp
| | | +-src/kernel32/pageManager.hpp
| | | +-src/kernel32/page.hpp
| | | +-src/kernel32/types.hpp
| | | +-/home/xaliver/BuildTools/cross/bin/x86_64-pc-linux-g++
| | +-build/kernel32/utils.o
| | | +-src/kernel32/utils.cpp
| | | +-src/kernel32/utils.hpp
| | | +-src/kernel32/types.hpp
| | | +-/home/xaliver/BuildTools/cross/bin/x86_64-pc-linux-g++
| | +-[build/kernel32/entry.bin]
| | +-/home/xaliver/BuildTools/cross/bin/x86_64-pc-linux-ld
| +-src/kernel32/main.cpp
| +-[build/kernel32/main.o]
| +-src/kernel32/memory.cpp
| +-[build/kernel32/memory.o]
| +-src/kernel32/pageManager.cpp
| +-[build/kernel32/pageManager.o]
| +-src/kernel32/utils.cpp
| +-[build/kernel32/utils.o]
+-src
+-src/SConscript
+-src/bootloader
| +-src/bootloader/BootLoader.asm
| +-src/bootloader/SConscript
+-src/kernel32
+-src/kernel32/SConscript
+-src/kernel32/asmUtils.asm
+-src/kernel32/cpu.cpp
+-src/kernel32/cpu.hpp
+-src/kernel32/entry.s
+-src/kernel32/main.cpp
+-src/kernel32/memory.cpp
+-src/kernel32/memory.hpp
+-src/kernel32/page.hpp
+-src/kernel32/pageManager.cpp
+-src/kernel32/pageManager.hpp
+-src/kernel32/types.hpp
+-src/kernel32/utils.cpp
+-src/kernel32/utils.hpp
scons: building terminated because of errors.

Related

How to prevent trimming of sbt dependencyTree output

Is it possible to configure sbt such that output from sbt dependencyTree is not trimmed?
Fragment of trimmed output:
[info] | | | | | | +-com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:2.12...
[info] | | | | | | +-com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:2.14.0
[info] | | | | | | | +-com.fasterxml.jackson.core:jackson-core:2.14.0 (evicted by: ..
[info] | | | | | | | +-com.fasterxml.jackson.core:jackson-core:2.14.1
[info] | | | | | | | +-com.fasterxml.jackson.core:jackson-databind:2.14.0 (evicted ..
[info] | | | | | | | +-com.fasterxml.jackson.core:jackson-databind:2.14.1
[info] | | | | | | | +-com.fasterxml.jackson.core:jackson-annotations:2.14.1
[info] | | | | | | | +-com.fasterxml.jackson.core:jackson-core:2.14.1
Trimming is problematic for me, as the output of this command is picked up by snyk cli, which gets confused and reports false positives.
I am using sbt 1.8.0
dependencyTree tasks observe asciiGraphWidth option starting with sbt 1.6.0.
See sbt 1.6.0 release notes
Thus:
add asciiGraphWidth is build.sbt:
asciiGraphWidth := 180
alternatively, this can be passed in via command line:
sbt 'set asciiGraphWidth := 180' dependencyTree

volume backup create what is errno 22?

Trying to create a volume backup both using the web UI and the cmd and keep getting errno 22. I'm unable to find information about the error or how to fix it. Anyone knows where I should start looking?
(openstack) volume backup create --force --name inventory01_vol_backups 398ee974-9b83-4918-9935-f52882b3e6b7
(openstack) volume backup show inventory01_vol_backups
+-----------------------+------------------------------------------------------------------+
| Field | Value |
+-----------------------+------------------------------------------------------------------+
| availability_zone | None |
| container | None |
| created_at | 2021-08-03T23:45:49.000000 |
| data_timestamp | 2021-08-03T23:45:49.000000 |
| description | None |
| fail_reason | [errno 22] RADOS invalid argument (error calling conf_read_file) |
| has_dependent_backups | False |
| id | 924c6e62-789e-4e51-9748-927695fc744c |
| is_incremental | False |
| name | inventory01_vol_backups |
| object_count | 0 |
| size | 30 |
| snapshot_id | None |
| status | error |
| updated_at | 2021-08-03T23:45:50.000000 |
| volume_id | 398ee974-9b83-4918-9935-f52882b3e6b7 |
+-----------------------+------------------------------------------------------------------+
The issue was caused due to a bug in Cinder version 16.2.1.dev13. Updating cinder to the latest version solved the issue

manipulate multiple variables in a data frame

How to shorten the following codes? I feel it's so repetitive and lengthy and perhaps can be shortened. Not sure how to select those variables and do the recoding like this in a succinct way. Any help is welcome!
data_France$X.1CTP2[data_France$X.1CTP2>7.01 | data_France$X.1CTP2<0.99]<-NA
data_France$X.1CTP3[data_France$X.1CTP3>7.01 | data_France$X.1CTP3<0.99]<-NA
data_France$X.1CTP4[data_France$X.1CTP4>7.01 | data_France$X.1CTP4<0.99]<-NA
data_France$X.1CTP5[data_France$X.1CTP5>7.01 | data_France$X.1CTP5<0.99]<-NA
data_France$X.1CTP6[data_France$X.1CTP6>7.01 | data_France$X.1CTP6<0.99]<-NA
data_France$X.1CTP7[data_France$X.1CTP7>7.01 | data_France$X.1CTP7<0.99]<-NA
data_France$X.1CTP8[data_France$X.1CTP8>7.01 | data_France$X.1CTP8<0.99]<-NA
data_France$X.1CTP9[data_France$X.1CTP9>7.01 | data_France$X.1CTP9<0.99]<-NA
data_France$X.1CTP10[data_France$X.1CTP10>7.01 | data_France$X.1CTP10<0.99]<-NA
data_France$X.1CTP11[data_France$X.1CTP11>7.01 | data_France$X.1CTP11<0.99]<-NA
data_France$X.1CTP12[data_France$X.1CTP12>7.01 | data_France$X.1CTP12<0.99]<-NA
data_France$X.1CTP13[data_France$X.1CTP13>7.01 | data_France$X.1CTP13<0.99]<-NA
data_France$X.1CTP14[data_France$X.1CTP14>7.01 | data_France$X.1CTP14<0.99]<-NA
data_France$X.1CTP15[data_France$X.1CTP15>7.01 | data_France$X.1CTP15<0.99]<-NA
data_France$X.2CTP1[data_France$X.2CTP1>7.01 | data_France$X.2CTP1<0.99]<-NA
data_France$X.2CTP3[data_France$X.2CTP3>7.01 | data_France$X.2CTP3<0.99]<-NA
data_France$X.2CTP4[data_France$X.2CTP4>7.01 | data_France$X.2CTP4<0.99]<-NA
data_France$X.2CTP5[data_France$X.2CTP5>7.01 | data_France$X.2CTP5<0.99]<-NA
data_France$X.2CTP6[data_France$X.2CTP6>7.01 | data_France$X.2CTP6<0.99]<-NA
data_France$X.2CTP7[data_France$X.2CTP7>7.01 | data_France$X.2CTP7<0.99]<-NA
data_France$X.2CTP8[data_France$X.2CTP8>7.01 | data_France$X.2CTP8<0.99]<-NA
data_France$X.2CTP9[data_France$X.2CTP9>7.01 | data_France$X.2CTP9<0.99]<-NA
data_France$X.2CTP10[data_France$X.2CTP10>7.01 | data_France$X.2CTP10<0.99]<-NA
data_France$X.2CTP11[data_France$X.2CTP11>7.01 | data_France$X.2CTP11<0.99]<-NA
data_France$X.2CTP12[data_France$X.2CTP12>7.01 | data_France$X.2CTP12<0.99]<-NA
data_France$X.2CTP13[data_France$X.2CTP13>7.01 | data_France$X.2CTP13<0.99]<-NA
data_France$X.2CTP14[data_France$X.2CTP14>7.01 | data_France$X.2CTP14<0.99]<-NA
data_France$X.2CTP15[data_France$X.2CTP15>7.01 | data_France$X.2CTP15<0.99]<-NA
data_France$X.3CTP1[data_France$X.3CTP1>7.01 | data_France$X.3CTP1<0.99]<-NA
data_France$X.3CTP2[data_France$X.3CTP2>7.01 | data_France$X.3CTP2<0.99]<-NA
data_France$X.3CTP4[data_France$X.3CTP4>7.01 | data_France$X.3CTP4<0.99]<-NA
data_France$X.3CTP5[data_France$X.3CTP5>7.01 | data_France$X.3CTP5<0.99]<-NA
data_France$X.3CTP6[data_France$X.3CTP6>7.01 | data_France$X.3CTP6<0.99]<-NA
data_France$X.3CTP7[data_France$X.3CTP7>7.01 | data_France$X.3CTP7<0.99]<-NA
data_France$X.3CTP8[data_France$X.3CTP8>7.01 | data_France$X.3CTP8<0.99]<-NA
data_France$X.3CTP9[data_France$X.3CTP9>7.01 | data_France$X.3CTP9<0.99]<-NA
data_France$X.3CTP10[data_France$X.3CTP10>7.01 | data_France$X.3CTP10<0.99]<-NA
data_France$X.3CTP11[data_France$X.3CTP11>7.01 | data_France$X.3CTP11<0.99]<-NA
data_France$X.3CTP12[data_France$X.3CTP12>7.01 | data_France$X.3CTP12<0.99]<-NA
data_France$X.3CTP13[data_France$X.3CTP13>7.01 | data_France$X.3CTP13<0.99]<-NA
data_France$X.3CTP14[data_France$X.3CTP14>7.01 | data_France$X.3CTP14<0.99]<-NA
data_France$X.3CTP15[data_France$X.3CTP15>7.01 | data_France$X.3CTP15<0.99]<-NA
data_France$X.4CTP1[data_France$X.4CTP1>7.01 | data_France$X.4CTP1<0.99]<-NA
data_France$X.4CTP2[data_France$X.4CTP2>7.01 | data_France$X.4CTP2<0.99]<-NA
data_France$X.4CTP3[data_France$X.4CTP3>7.01 | data_France$X.4CTP3<0.99]<-NA
data_France$X.4CTP5[data_France$X.4CTP5>7.01 | data_France$X.4CTP5<0.99]<-NA
data_France$X.4CTP6[data_France$X.4CTP6>7.01 | data_France$X.4CTP6<0.99]<-NA
data_France$X.4CTP7[data_France$X.4CTP7>7.01 | data_France$X.4CTP7<0.99]<-NA
data_France$X.4CTP8[data_France$X.4CTP8>7.01 | data_France$X.4CTP8<0.99]<-NA
data_France$X.4CTP9[data_France$X.4CTP9>7.01 | data_France$X.4CTP9<0.99]<-NA
data_France$X.4CTP10[data_France$X.4CTP10>7.01 | data_France$X.4CTP10<0.99]<-NA
data_France$X.4CTP11[data_France$X.4CTP11>7.01 | data_France$X.4CTP11<0.99]<-NA
data_France$X.4CTP12[data_France$X.4CTP12>7.01 | data_France$X.4CTP12<0.99]<-NA
data_France$X.4CTP13[data_France$X.4CTP13>7.01 | data_France$X.4CTP13<0.99]<-NA
data_France$X.4CTP14[data_France$X.4CTP14>7.01 | data_France$X.4CTP14<0.99]<-NA
data_France$X.4CTP15[data_France$X.4CTP15>7.01 | data_France$X.4CTP15<0.99]<-NA
data_France$X.5CTP1[data_France$X.5CTP1>7.01 | data_France$X.5CTP1<0.99]<-NA
data_France$X.5CTP2[data_France$X.5CTP2>7.01 | data_France$X.5CTP2<0.99]<-NA
data_France$X.5CTP3[data_France$X.5CTP3>7.01 | data_France$X.5CTP3<0.99]<-NA
data_France$X.5CTP4[data_France$X.5CTP4>7.01 | data_France$X.5CTP4<0.99]<-NA
data_France$X.5CTP6[data_France$X.5CTP6>7.01 | data_France$X.5CTP6<0.99]<-NA
data_France$X.5CTP7[data_France$X.5CTP7>7.01 | data_France$X.5CTP7<0.99]<-NA
data_France$X.5CTP8[data_France$X.5CTP8>7.01 | data_France$X.5CTP8<0.99]<-NA
data_France$X.5CTP9[data_France$X.5CTP9>7.01 | data_France$X.5CTP9<0.99]<-NA
data_France$X.5CTP10[data_France$X.5CTP10>7.01 | data_France$X.5CTP10<0.99]<-NA
data_France$X.5CTP11[data_France$X.5CTP11>7.01 | data_France$X.5CTP11<0.99]<-NA
data_France$X.5CTP12[data_France$X.5CTP12>7.01 | data_France$X.5CTP12<0.99]<-NA
data_France$X.5CTP13[data_France$X.5CTP13>7.01 | data_France$X.5CTP13<0.99]<-NA
data_France$X.5CTP14[data_France$X.5CTP14>7.01 | data_France$X.5CTP14<0.99]<-NA
data_France$X.5CTP15[data_France$X.5CTP15>7.01 | data_France$X.5CTP15<0.99]<-NA
data_France$X.6CTP1[data_France$X.6CTP1>7.01 | data_France$X.6CTP1<0.99]<-NA
data_France$X.6CTP2[data_France$X.6CTP2>7.01 | data_France$X.6CTP2<0.99]<-NA
data_France$X.6CTP3[data_France$X.6CTP3>7.01 | data_France$X.6CTP3<0.99]<-NA
data_France$X.6CTP4[data_France$X.6CTP4>7.01 | data_France$X.6CTP4<0.99]<-NA
data_France$X.6CTP5[data_France$X.6CTP5>7.01 | data_France$X.6CTP5<0.99]<-NA
data_France$X.6CTP7[data_France$X.6CTP7>7.01 | data_France$X.6CTP7<0.99]<-NA
data_France$X.6CTP8[data_France$X.6CTP8>7.01 | data_France$X.6CTP8<0.99]<-NA
data_France$X.6CTP9[data_France$X.6CTP9>7.01 | data_France$X.6CTP9<0.99]<-NA
data_France$X.6CTP10[data_France$X.6CTP10>7.01 | data_France$X.6CTP10<0.99]<-NA
data_France$X.6CTP11[data_France$X.6CTP11>7.01 | data_France$X.6CTP11<0.99]<-NA
data_France$X.6CTP12[data_France$X.6CTP12>7.01 | data_France$X.6CTP12<0.99]<-NA
data_France$X.6CTP13[data_France$X.6CTP13>7.01 | data_France$X.6CTP13<0.99]<-NA
data_France$X.6CTP14[data_France$X.6CTP14>7.01 | data_France$X.6CTP14<0.99]<-NA
data_France$X.6CTP15[data_France$X.6CTP15>7.01 | data_France$X.6CTP15<0.99]<-NA
data_France$X.7CTP1[data_France$X.7CTP1>7.01 | data_France$X.7CTP1<0.99]<-NA
data_France$X.7CTP2[data_France$X.7CTP2>7.01 | data_France$X.7CTP2<0.99]<-NA
data_France$X.7CTP3[data_France$X.7CTP3>7.01 | data_France$X.7CTP3<0.99]<-NA
data_France$X.7CTP4[data_France$X.7CTP4>7.01 | data_France$X.7CTP4<0.99]<-NA
data_France$X.7CTP5[data_France$X.7CTP5>7.01 | data_France$X.7CTP5<0.99]<-NA
data_France$X.7CTP6[data_France$X.7CTP6>7.01 | data_France$X.7CTP6<0.99]<-NA
data_France$X.7CTP8[data_France$X.7CTP8>7.01 | data_France$X.7CTP8<0.99]<-NA
data_France$X.7CTP9[data_France$X.7CTP9>7.01 | data_France$X.7CTP9<0.99]<-NA
data_France$X.7CTP10[data_France$X.7CTP10>7.01 | data_France$X.7CTP10<0.99]<-NA
data_France$X.7CTP11[data_France$X.7CTP11>7.01 | data_France$X.7CTP11<0.99]<-NA
data_France$X.7CTP12[data_France$X.7CTP12>7.01 | data_France$X.7CTP12<0.99]<-NA
data_France$X.7CTP13[data_France$X.7CTP13>7.01 | data_France$X.7CTP13<0.99]<-NA
data_France$X.7CTP14[data_France$X.7CTP14>7.01 | data_France$X.7CTP14<0.99]<-NA
data_France$X.7CTP15[data_France$X.7CTP15>7.01 | data_France$X.7CTP15<0.99]<-NA
data_France$X.8CTP1[data_France$X.8CTP1>7.01 | data_France$X.8CTP1<0.99]<-NA
data_France$X.8CTP2[data_France$X.8CTP2>7.01 | data_France$X.8CTP2<0.99]<-NA
data_France$X.8CTP3[data_France$X.8CTP3>7.01 | data_France$X.8CTP3<0.99]<-NA
data_France$X.8CTP4[data_France$X.8CTP4>7.01 | data_France$X.8CTP4<0.99]<-NA
data_France$X.8CTP5[data_France$X.8CTP5>7.01 | data_France$X.8CTP5<0.99]<-NA
data_France$X.8CTP6[data_France$X.8CTP6>7.01 | data_France$X.8CTP6<0.99]<-NA
data_France$X.8CTP7[data_France$X.8CTP7>7.01 | data_France$X.8CTP7<0.99]<-NA
data_France$X.8CTP9[data_France$X.8CTP9>7.01 | data_France$X.8CTP9<0.99]<-NA
data_France$X.8CTP10[data_France$X.8CTP10>7.01 | data_France$X.8CTP10<0.99]<-NA
data_France$X.8CTP11[data_France$X.8CTP11>7.01 | data_France$X.8CTP11<0.99]<-NA
data_France$X.8CTP12[data_France$X.8CTP12>7.01 | data_France$X.8CTP12<0.99]<-NA
data_France$X.8CTP13[data_France$X.8CTP13>7.01 | data_France$X.8CTP13<0.99]<-NA
data_France$X.8CTP14[data_France$X.8CTP14>7.01 | data_France$X.8CTP14<0.99]<-NA
data_France$X.8CTP15[data_France$X.8CTP15>7.01 | data_France$X.8CTP15<0.99]<-NA
data_France$X.9CTP1[data_France$X.9CTP1>7.01 | data_France$X.9CTP1<0.99]<-NA
data_France$X.9CTP2[data_France$X.9CTP2>7.01 | data_France$X.9CTP2<0.99]<-NA
data_France$X.9CTP3[data_France$X.9CTP3>7.01 | data_France$X.9CTP3<0.99]<-NA
data_France$X.9CTP4[data_France$X.9CTP4>7.01 | data_France$X.9CTP4<0.99]<-NA
data_France$X.9CTP5[data_France$X.9CTP5>7.01 | data_France$X.9CTP5<0.99]<-NA
data_France$X.9CTP6[data_France$X.9CTP6>7.01 | data_France$X.9CTP6<0.99]<-NA
data_France$X.9CTP7[data_France$X.9CTP7>7.01 | data_France$X.9CTP7<0.99]<-NA
data_France$X.9CTP8[data_France$X.9CTP8>7.01 | data_France$X.9CTP8<0.99]<-NA
data_France$X.9CTP10[data_France$X.9CTP10>7.01 | data_France$X.9CTP10<0.99]<-NA
data_France$X.9CTP11[data_France$X.9CTP11>7.01 | data_France$X.9CTP11<0.99]<-NA
data_France$X.9CTP12[data_France$X.9CTP12>7.01 | data_France$X.9CTP12<0.99]<-NA
data_France$X.9CTP13[data_France$X.9CTP13>7.01 | data_France$X.9CTP13<0.99]<-NA
data_France$X.9CTP14[data_France$X.9CTP14>7.01 | data_France$X.9CTP14<0.99]<-NA
data_France$X.9CTP15[data_France$X.9CTP15>7.01 | data_France$X.9CTP15<0.99]<-NA
data_France$X.10CTP1[data_France$X.10CTP1>7.01 | data_France$X.10CTP1<0.99]<-NA
data_France$X.10CTP2[data_France$X.10CTP2>7.01 | data_France$X.10CTP2<0.99]<-NA
data_France$X.10CTP3[data_France$X.10CTP3>7.01 | data_France$X.10CTP3<0.99]<-NA
data_France$X.10CTP4[data_France$X.10CTP4>7.01 | data_France$X.10CTP4<0.99]<-NA
data_France$X.10CTP5[data_France$X.10CTP5>7.01 | data_France$X.10CTP5<0.99]<-NA
data_France$X.10CTP6[data_France$X.10CTP6>7.01 | data_France$X.10CTP6<0.99]<-NA
data_France$X.10CTP7[data_France$X.10CTP7>7.01 | data_France$X.10CTP7<0.99]<-NA
data_France$X.10CTP8[data_France$X.10CTP8>7.01 | data_France$X.10CTP8<0.99]<-NA
data_France$X.10CTP9[data_France$X.10CTP9>7.01 | data_France$X.10CTP9<0.99]<-NA
data_France$X.10CTP11[data_France$X.10CTP11>7.01 | data_France$X.10CTP11<0.99]<-NA
data_France$X.10CTP12[data_France$X.10CTP12>7.01 | data_France$X.10CTP12<0.99]<-NA
data_France$X.10CTP13[data_France$X.10CTP13>7.01 | data_France$X.10CTP13<0.99]<-NA
data_France$X.10CTP14[data_France$X.10CTP14>7.01 | data_France$X.10CTP14<0.99]<-NA
data_France$X.10CTP15[data_France$X.10CTP15>7.01 | data_France$X.10CTP15<0.99]<-NA
data_France$X.11CTP1[data_France$X.11CTP1>7.01 | data_France$X.11CTP1<0.99]<-NA
data_France$X.11CTP2[data_France$X.11CTP2>7.01 | data_France$X.11CTP2<0.99]<-NA
data_France$X.11CTP3[data_France$X.11CTP3>7.01 | data_France$X.11CTP3<0.99]<-NA
data_France$X.11CTP4[data_France$X.11CTP4>7.01 | data_France$X.11CTP4<0.99]<-NA
data_France$X.11CTP5[data_France$X.11CTP5>7.01 | data_France$X.11CTP5<0.99]<-NA
data_France$X.11CTP6[data_France$X.11CTP6>7.01 | data_France$X.11CTP6<0.99]<-NA
data_France$X.11CTP7[data_France$X.11CTP7>7.01 | data_France$X.11CTP7<0.99]<-NA
data_France$X.11CTP8[data_France$X.11CTP8>7.01 | data_France$X.11CTP8<0.99]<-NA
data_France$X.11CTP9[data_France$X.11CTP9>7.01 | data_France$X.11CTP9<0.99]<-NA
data_France$X.11CTP10[data_France$X.11CTP10>7.01 | data_France$X.11CTP10<0.99]<-NA
data_France$X.11CTP12[data_France$X.11CTP12>7.01 | data_France$X.11CTP12<0.99]<-NA
data_France$X.11CTP13[data_France$X.11CTP13>7.01 | data_France$X.11CTP13<0.99]<-NA
data_France$X.11CTP14[data_France$X.11CTP14>7.01 | data_France$X.11CTP14<0.99]<-NA
data_France$X.11CTP15[data_France$X.11CTP15>7.01 | data_France$X.11CTP15<0.99]<-NA
data_France$X.12CTP1[data_France$X.12CTP1>7.01 | data_France$X.12CTP1<0.99]<-NA
data_France$X.12CTP2[data_France$X.12CTP2>7.01 | data_France$X.12CTP2<0.99]<-NA
data_France$X.12CTP3[data_France$X.12CTP3>7.01 | data_France$X.12CTP3<0.99]<-NA
data_France$X.12CTP4[data_France$X.12CTP4>7.01 | data_France$X.12CTP4<0.99]<-NA
data_France$X.12CTP5[data_France$X.12CTP5>7.01 | data_France$X.12CTP5<0.99]<-NA
data_France$X.12CTP6[data_France$X.12CTP6>7.01 | data_France$X.12CTP6<0.99]<-NA
data_France$X.12CTP7[data_France$X.12CTP7>7.01 | data_France$X.12CTP7<0.99]<-NA
data_France$X.12CTP8[data_France$X.12CTP8>7.01 | data_France$X.12CTP8<0.99]<-NA
data_France$X.12CTP9[data_France$X.12CTP9>7.01 | data_France$X.12CTP9<0.99]<-NA
data_France$X.12CTP10[data_France$X.12CTP10>7.01 | data_France$X.12CTP10<0.99]<-NA
data_France$X.12CTP11[data_France$X.12CTP11>7.01 | data_France$X.12CTP11<0.99]<-NA
data_France$X.12CTP13[data_France$X.12CTP13>7.01 | data_France$X.12CTP13<0.99]<-NA
data_France$X.12CTP14[data_France$X.12CTP14>7.01 | data_France$X.12CTP14<0.99]<-NA
data_France$X.12CTP15[data_France$X.12CTP15>7.01 | data_France$X.12CTP15<0.99]<-NA
data_France$X.13CTP1[data_France$X.13CTP1>7.01 | data_France$X.13CTP1<0.99]<-NA
data_France$X.13CTP2[data_France$X.13CTP2>7.01 | data_France$X.13CTP2<0.99]<-NA
data_France$X.13CTP3[data_France$X.13CTP3>7.01 | data_France$X.13CTP3<0.99]<-NA
data_France$X.13CTP4[data_France$X.13CTP4>7.01 | data_France$X.13CTP4<0.99]<-NA
data_France$X.13CTP5[data_France$X.13CTP5>7.01 | data_France$X.13CTP5<0.99]<-NA
data_France$X.13CTP6[data_France$X.13CTP6>7.01 | data_France$X.13CTP6<0.99]<-NA
data_France$X.13CTP7[data_France$X.13CTP7>7.01 | data_France$X.13CTP7<0.99]<-NA
data_France$X.13CTP8[data_France$X.13CTP8>7.01 | data_France$X.13CTP8<0.99]<-NA
data_France$X.13CTP9[data_France$X.13CTP9>7.01 | data_France$X.13CTP9<0.99]<-NA
data_France$X.13CTP10[data_France$X.13CTP10>7.01 | data_France$X.13CTP10<0.99]<-NA
data_France$X.13CTP11[data_France$X.13CTP11>7.01 | data_France$X.13CTP11<0.99]<-NA
data_France$X.13CTP12[data_France$X.13CTP12>7.01 | data_France$X.13CTP12<0.99]<-NA
data_France$X.13CTP14[data_France$X.13CTP14>7.01 | data_France$X.13CTP14<0.99]<-NA
data_France$X.13CTP15[data_France$X.13CTP15>7.01 | data_France$X.13CTP15<0.99]<-NA
data_France$X.14CTP1[data_France$X.14CTP1>7.01 | data_France$X.14CTP1<0.99]<-NA
data_France$X.14CTP2[data_France$X.14CTP2>7.01 | data_France$X.14CTP2<0.99]<-NA
data_France$X.14CTP3[data_France$X.14CTP3>7.01 | data_France$X.14CTP3<0.99]<-NA
data_France$X.14CTP4[data_France$X.14CTP4>7.01 | data_France$X.14CTP4<0.99]<-NA
data_France$X.14CTP5[data_France$X.14CTP5>7.01 | data_France$X.14CTP5<0.99]<-NA
data_France$X.14CTP6[data_France$X.14CTP6>7.01 | data_France$X.14CTP6<0.99]<-NA
data_France$X.14CTP7[data_France$X.14CTP7>7.01 | data_France$X.14CTP7<0.99]<-NA
data_France$X.14CTP8[data_France$X.14CTP8>7.01 | data_France$X.14CTP8<0.99]<-NA
data_France$X.14CTP9[data_France$X.14CTP9>7.01 | data_France$X.14CTP9<0.99]<-NA
data_France$X.14CTP10[data_France$X.14CTP10>7.01 | data_France$X.14CTP10<0.99]<-NA
data_France$X.14CTP11[data_France$X.14CTP11>7.01 | data_France$X.14CTP11<0.99]<-NA
data_France$X.14CTP12[data_France$X.14CTP12>7.01 | data_France$X.14CTP12<0.99]<-NA
data_France$X.14CTP13[data_France$X.14CTP13>7.01 | data_France$X.14CTP13<0.99]<-NA
data_France$X.14CTP15[data_France$X.14CTP15>7.01 | data_France$X.14CTP15<0.99]<-NA
data_France$X.15CTP1[data_France$X.15CTP1>7.01 | data_France$X.15CTP1<0.99]<-NA
data_France$X.15CTP2[data_France$X.15CTP2>7.01 | data_France$X.15CTP2<0.99]<-NA
data_France$X.15CTP3[data_France$X.15CTP3>7.01 | data_France$X.15CTP3<0.99]<-NA
data_France$X.15CTP4[data_France$X.15CTP4>7.01 | data_France$X.15CTP4<0.99]<-NA
data_France$X.15CTP5[data_France$X.15CTP5>7.01 | data_France$X.15CTP5<0.99]<-NA
data_France$X.15CTP6[data_France$X.15CTP6>7.01 | data_France$X.15CTP6<0.99]<-NA
data_France$X.15CTP7[data_France$X.15CTP7>7.01 | data_France$X.15CTP7<0.99]<-NA
data_France$X.15CTP8[data_France$X.15CTP8>7.01 | data_France$X.15CTP8<0.99]<-NA
data_France$X.15CTP9[data_France$X.15CTP9>7.01 | data_France$X.15CTP9<0.99]<-NA
data_France$X.15CTP10[data_France$X.15CTP10>7.01 | data_France$X.15CTP10<0.99]<-NA
data_France$X.15CTP11[data_France$X.15CTP11>7.01 | data_France$X.15CTP11<0.99]<-NA
data_France$X.15CTP12[data_France$X.15CTP12>7.01 | data_France$X.15CTP12<0.99]<-NA
data_France$X.15CTP13[data_France$X.15CTP13>7.01 | data_France$X.15CTP13<0.99]<-NA
data_France$X.15CTP14[data_France$X.15CTP14>7.01 | data_France$X.15CTP14<0.99]<-NA
Base R equivalent of #Cettt's answer:
## helper function to replace elements with NA
rfun <- function(x) replace(x, which(x<0.99 | x>7.01), NA)
## identify which columns need to be changed
cnm <- grep("^X.[0-9]+CTP[0-9]+", names(data_France))
for (i in cnm) {
data_France[cnm] <- rfun(data_France[cnm])
}
You could also use lapply(), but sometimes the for loop is easier to understand and debug.
I would recommend the dplyr package which has the mutate_at function.
In your case you could use it like this:
library(dplyr)
data_France %>%
as_tibble %>%
mutate_at(vars(matches("^X.[0-9]+CTP[0-9]+")), ~ifelse(.x < 0.99 | .x > 7.01, NA_real_, .x))
#Create a vector of variable names. There may be other ways to do this, like using
#regex or just taking the indices of the variables names (e.g., 1:225)
vars <- apply(expand.grid("X.", as.character(1:15), "CTP", as.character(1:15)),
1, paste0, collapse = "")
for (i in vars) {
data_France[[i]][data_France[[i]] > 7.01 | data_France[[i]] < 0.99] <- NA
}
If this is your entire data set (i.e., there are no other variables in the data), you can simply do
data_France[data_France > 7.01 | data_France < 0.99] <- NA

R: Regex to match more than one pipe occurrence

I have a dataset in which I paste values in a dplyr chain and collapse with the pipe character (e.g. " | "). If any of the values in the dataset are blank, I just get recurring pipe characters in the pasted list.
Some of the values look like this, for example:
badstring = "| | | | | | GHOULSBY,SCROGGINS | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | CAT,JOHNSON | | | | | | | | | | | | BURGLAR,PALA | | | | | | | | |"
I want to match all the pipes that occur more than once and delete them, so that just the names appear like so:
correctstring = "| GHOULSBY,SCROGGINS | CAT,JOHNSON | |BURGLAR,PALA |"
I tried the following, but to no avail:
mutate(names = gsub('[\\|]{2,}', '', name_list))
The difficulty in this question is in formulating a regex which can selectively remove every pipe, except the ones we want to remain as actual separators between terms. We can match on the following pattern:
\|\s+(?=\|)
and then replace just empty string. This pattern will remove any pipe (and any following whitespace) so long as what follows is another pipe. A removal would not occur when a pipe is followed by an actual term, or when it is followed by the end of the string.
badstring = "| | | | | | GHOULSBY,SCROGGINS | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | CAT,JOHNSON | | | | | | | | | | | | BURGLAR,PALA | | | | | | | | |"
result <- gsub("\\|\\s+(?=\\|)", "", badstring, perl=TRUE)
result
[1] "| GHOULSBY,SCROGGINS | CAT,JOHNSON | BURGLAR,PALA |"
Demo
Edit:
If you expect to have inputs like | | | which are devoid of any terms, and you would expect empty string as the output, then my solution would fail. I don't see an obvious way to modify the above regex, but you can handle this case with one more call to sub:
result <- sub("^\\|$", "", result)
We also might be able to modify the original pattern to use an alternation covering all cases:
result <- gsub("\\|\\s+(?=\\|)|(?:^\\|$)", "", badstring, perl=TRUE)

Unable to forecast linear model in R

I'm able to do forecasts with an ARIMA model, but when I try to do a forecast for a linear model, I do not get any actual forecasts - it stops at the end of the data set (which isn't useful for forecasting since I already know what's in the data set). I've found countless examples online where using this same code works just fine, but I haven't found anyone else having this same error.
library("stats")
library("forecast")
y <- data$Mfg.Shipments.Total..USA.
model_a1 <- auto.arima(y)
forecast_a1 <- forecast.Arima(model_a1, h = 12)
The above code works perfectly. However, when I try to do a linear model....
model1 <- lm(y ~ Mfg.NO.Total..USA. + Mfg.Inv.Total..USA., data = data )
f1 <- forecast.lm(model1, h = 12)
I get an error message saying that I MUST provide a new data set (which seems odd to me, since the documentation for the forecast package says that it is an optional argument).
f1 <- forecast.lm(model1, newdata = x, h = 12)
If I do this, I am able to get the function to work, but the forecast only predicts values for the existing data - it doesn't predict the next 12 periods. I have also tried using the append function to add additional rows to see if that would fix the issue, but when trying to forecast a linear model, it immediately stops at the most recent point in the time series.
Here's the data that I'm using:
+------------+---------------------------+--------------------+---------------------+
| | Mfg.Shipments.Total..USA. | Mfg.NO.Total..USA. | Mfg.Inv.Total..USA. |
+------------+---------------------------+--------------------+---------------------+
| 2110-01-01 | 3.59746e+11 | 3.58464e+11 | 5.01361e+11 |
| 2110-01-01 | 3.59746e+11 | 3.58464e+11 | 5.01361e+11 |
| 2110-02-01 | 3.62268e+11 | 3.63441e+11 | 5.10439e+11 |
| 2110-03-01 | 4.23748e+11 | 4.24527e+11 | 5.10792e+11 |
| 2110-04-01 | 4.08755e+11 | 4.02769e+11 | 5.16853e+11 |
| 2110-05-01 | 4.08187e+11 | 4.02869e+11 | 5.18180e+11 |
| 2110-06-01 | 4.27567e+11 | 4.21713e+11 | 5.15675e+11 |
| 2110-07-01 | 3.97590e+11 | 3.89916e+11 | 5.24785e+11 |
| 2110-08-01 | 4.24732e+11 | 4.16304e+11 | 5.27734e+11 |
| 2110-09-01 | 4.30974e+11 | 4.35043e+11 | 5.28797e+11 |
| 2110-10-01 | 4.24008e+11 | 4.17076e+11 | 5.38917e+11 |
| 2110-11-01 | 4.11930e+11 | 4.09440e+11 | 5.42618e+11 |
| 2110-12-01 | 4.25940e+11 | 4.34201e+11 | 5.35384e+11 |
| 2111-01-01 | 4.01629e+11 | 4.07748e+11 | 5.55057e+11 |
| 2111-02-01 | 4.06385e+11 | 4.06151e+11 | 5.66058e+11 |
| 2111-03-01 | 4.83827e+11 | 4.89904e+11 | 5.70990e+11 |
| 2111-04-01 | 4.54640e+11 | 4.46702e+11 | 5.84808e+11 |
| 2111-05-01 | 4.65124e+11 | 4.63155e+11 | 5.92456e+11 |
| 2111-06-01 | 4.83809e+11 | 4.75150e+11 | 5.86645e+11 |
| 2111-07-01 | 4.44437e+11 | 4.40452e+11 | 5.97201e+11 |
| 2111-08-01 | 4.83537e+11 | 4.79958e+11 | 5.99461e+11 |
| 2111-09-01 | 4.77130e+11 | 4.75580e+11 | 5.93065e+11 |
| 2111-10-01 | 4.69276e+11 | 4.59579e+11 | 6.03481e+11 |
| 2111-11-01 | 4.53706e+11 | 4.55029e+11 | 6.02577e+11 |
| 2111-12-01 | 4.57872e+11 | 4.81454e+11 | 5.86886e+11 |
| 2112-01-01 | 4.35834e+11 | 4.45037e+11 | 6.04042e+11 |
| 2112-02-01 | 4.55996e+11 | 4.70820e+11 | 6.12071e+11 |
| 2112-03-01 | 5.04869e+11 | 5.08818e+11 | 6.11717e+11 |
| 2112-04-01 | 4.76213e+11 | 4.70666e+11 | 6.16375e+11 |
| 2112-05-01 | 4.95789e+11 | 4.87730e+11 | 6.17639e+11 |
| 2112-06-01 | 4.91218e+11 | 4.87857e+11 | 6.09361e+11 |
| 2112-07-01 | 4.58087e+11 | 4.61037e+11 | 6.19166e+11 |
| 2112-08-01 | 4.97438e+11 | 4.74539e+11 | 6.22773e+11 |
| 2112-09-01 | 4.86994e+11 | 4.85560e+11 | 6.23067e+11 |
| 2112-10-01 | 4.96744e+11 | 4.92562e+11 | 6.26796e+11 |
| 2112-11-01 | 4.70810e+11 | 4.64944e+11 | 6.23999e+11 |
| 2112-12-01 | 4.66721e+11 | 4.88615e+11 | 6.08900e+11 |
| 2113-01-01 | 4.51585e+11 | 4.50763e+11 | 6.25881e+11 |
| 2113-02-01 | 4.56329e+11 | 4.69574e+11 | 6.33157e+11 |
| 2113-03-01 | 5.04023e+11 | 4.92978e+11 | 6.31055e+11 |
| 2113-04-01 | 4.84798e+11 | 4.76750e+11 | 6.35643e+11 |
| 2113-05-01 | 5.04478e+11 | 5.04488e+11 | 6.34376e+11 |
| 2113-06-01 | 4.99043e+11 | 5.13760e+11 | 6.25715e+11 |
| 2113-07-01 | 4.75700e+11 | 4.69012e+11 | 6.34892e+11 |
| 2113-08-01 | 5.05244e+11 | 4.90404e+11 | 6.37735e+11 |
| 2113-09-01 | 5.00087e+11 | 5.04849e+11 | 6.34665e+11 |
| 2113-10-01 | 5.05965e+11 | 4.99682e+11 | 6.38945e+11 |
| 2113-11-01 | 4.78876e+11 | 4.80784e+11 | 6.34442e+11 |
| 2113-12-01 | 4.80640e+11 | 4.98807e+11 | 6.19458e+11 |
| 2114-01-01 | 4.56779e+11 | 4.57684e+11 | 6.36568e+11 |
| 2114-02-01 | 4.62195e+11 | 4.70312e+11 | 6.48982e+11 |
| 2114-03-01 | 5.19472e+11 | 5.25900e+11 | 6.47038e+11 |
| 2114-04-01 | 5.04217e+11 | 5.06090e+11 | 6.52612e+11 |
| 2114-05-01 | 5.14186e+11 | 5.11149e+11 | 6.58990e+11 |
| 2114-06-01 | 5.25249e+11 | 5.33247e+11 | 6.49512e+11 |
| 2114-07-01 | 4.99198e+11 | 5.52506e+11 | 6.57645e+11 |
| 2114-08-01 | 5.17184e+11 | 5.07622e+11 | 6.59281e+11 |
| 2114-09-01 | 5.23682e+11 | 5.24051e+11 | 6.55582e+11 |
| 2114-10-01 | 5.17305e+11 | 5.09549e+11 | 6.59237e+11 |
| 2114-11-01 | 4.71921e+11 | 4.70093e+11 | 6.57044e+11 |
| 2114-12-01 | 4.84948e+11 | 4.86804e+11 | 6.34120e+11 |
+------------+---------------------------+--------------------+---------------------+
Edit - Here's the code I used for adding new datapoints for forecasting.
library(xts)
library(mondate)
d <- as.mondate("2115-01-01")
d11 <- d + 11
seq(d, d11)
newdates <- seq(d, d11)
new_xts <- xts(order.by = as.Date(newdates))
new_xts$Mfg.Shipments.Total..USA. <- NA
new_xts$Mfg.NO.Total..USA. <- NA
new_xts$Mfg.Inv.Total..USA. <- NA
x <- append(data, new_xts)
Not sure if you ever figured this out, but just in case I thought I'd point out what's going wrong.
The documentation for forecast.lm says:
An optional data frame in which to look for variables with which to predict. If omitted, it is assumed that the only variables are trend and season, and h forecasts are produced.
so it's optional if trend and season are your only predictors.
The ARIMA model works because it's using lagged values of the time series in the forecast. For the linear model, it uses the given predictors (Mfg.NO.Total..USA. and Mfg.Inv.Total..USA. in your case) and thus needs their corresponding future values; without these, there are no independent variables to predict from.
In the edit, you added those variables to your future dataset, but they still have values of NA for all future points, thus the forecasts are also NA.
Gabe is correct. You need future values of your causals.
You should consider the Transfer Function modeling process instead of regression(ie developed for use with cross-sectional data). By using prewhitening your X variables (ie build a model for each one), you can calculate the Cross correlation function to see any lead or lag relationship.
It is very apparent that Inv.Total is a lead variable(b**-1) from the standardized graph of Y and the two x's. When Invto moves down so does shipments. In addition, there is also AR seasonal component beyond the causals that is driving the data. There are a few outliers as well so this is a robust solution. I am developer of this software used here, but this can be run in any tool.

Resources