Tokenizers 설치/에러/해결

https://yunny-p.tistory.com/entry/Tokenizers-%EC%84%A4%EC%B9%98

Tokenizers 설치/에러/해결

 

---

Tokenizers 설치/에러/해결

pip로 설치하면 다음과 같다

pip install tokenizers

하지만 설치가 안될 경우가 종종 있다.

[코드를 다운 받아서 설치하는 방법]

-anaconda에 설치하기 때문에 새로운 방법이 필요했다.

먼저 rust를 설치한다.

curl https://sh.rustup.rs -sSf | sh -s -- -y
export PATH="$HOME/.cargo/bin:$PATH"

다음에는 코드를 다운받아서 설치한다.

git clone https://github.com/huggingface/tokenizers
cd tokenizers/bindings/python

# Install `tokenizers` 
conda install -c conda-forge setuptools-rust
python setup.py install

conda를 사용하여 설치하기 때문에 pip install setuptools_rust 대신에 conda install -c conda-forge setuptools-rust를 사용한다.

이렇게 하면 될 것 같았는데 conda라서 그런지 conda env에 설치되지 않았다. 

따라서  ./bindings/python/build/lib.linux-x86_64-3.7/tokenizers의 파일을 conda의 가상환경 위치(~/anaconda3/envs/pl/lib/python3.7/site-packages/)로 cp -r 를 이용하여 옮겨놓았더니 해결

 

 

---

내가 쓴 해결법

curl https://sh.rustup.rs -sSf | sh -s -- -y
export PATH="$HOME/.cargo/bin:$PATH"



git clone https://github.com/huggingface/tokenizers
cd tokenizers/bindings/python


# Install `tokenizers` 
pip install setuptools_rust 
python setup.py install

나는 아나콘다 안쓰므로 위와 같이 했다

python setup.py install에서는 오류가 났지만 이후 

 

pip install tokenizers 를 하니 설치가 됐다.

 

아래는 설치 진행과정

sonsangwoo@sonui-Macmini nia_project % curl https://sh.rustup.rs -sSf | sh -s -- -y
export PATH="$HOME/.cargo/bin:$PATH"
info: downloading installer
info: profile set to 'default'
info: default host triple is aarch64-apple-darwin
warning: Updating existing toolchain, profile choice will be ignored
info: syncing channel updates for 'stable-aarch64-apple-darwin'
info: default toolchain set to 'stable-aarch64-apple-darwin'

  stable-aarch64-apple-darwin unchanged - rustc 1.61.0 (fe5b13d68 2022-05-18)


Rust is installed now. Great!

To get started you may need to restart your current shell.
This would reload your PATH environment variable to include
Cargo's bin directory ($HOME/.cargo/bin).

To configure your current shell, run:
source $HOME/.cargo/env
sonsangwoo@sonui-Macmini nia_project % git clone https://github.com/huggingface/tokenizers
cd tokenizers/bindings/python
'tokenizers'에 복제합니다...
remote: Enumerating objects: 15323, done.
remote: Counting objects: 100% (2081/2081), done.
remote: Compressing objects: 100% (958/958), done.
remote: Total 15323 (delta 1368), reused 1648 (delta 1088), pack-reused 13242
오브젝트를 받는 중: 100% (15323/15323), 7.59 MiB | 10.69 MiB/s, 완료.
델타를 알아내는 중: 100% (9999/9999), 완료.
sonsangwoo@sonui-Macmini python % cd tokenizers/bindings/python
cd: no such file or directory: tokenizers/bindings/python
sonsangwoo@sonui-Macmini python % pip install setuptools_rust
Collecting setuptools_rust
  Using cached setuptools_rust-1.3.0-py3-none-any.whl (21 kB)
Requirement already satisfied: setuptools>=58.0 in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from setuptools_rust) (62.3.2)
Requirement already satisfied: typing-extensions>=3.7.4.3 in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from setuptools_rust) (4.2.0)
Collecting semantic-version<3,>=2.8.2
  Using cached semantic_version-2.9.0-py2.py3-none-any.whl (15 kB)
Installing collected packages: semantic-version, setuptools_rust
Successfully installed semantic-version-2.9.0 setuptools_rust-1.3.0
sonsangwoo@sonui-Macmini python % python setup.py install
Traceback (most recent call last):
  File "setup.py", line 2, in <module>
    from setuptools_rust import Binding, RustExtension
ImportError: No module named setuptools_rust
sonsangwoo@sonui-Macmini python % pip install setuptools_rust
Requirement already satisfied: setuptools_rust in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (1.3.0)
Requirement already satisfied: setuptools>=58.0 in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from setuptools_rust) (62.3.2)
Requirement already satisfied: typing-extensions>=3.7.4.3 in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from setuptools_rust) (4.2.0)
Requirement already satisfied: semantic-version<3,>=2.8.2 in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from setuptools_rust) (2.9.0)
sonsangwoo@sonui-Macmini python % python setup.py install    
Traceback (most recent call last):
  File "setup.py", line 2, in <module>
    from setuptools_rust import Binding, RustExtension
ImportError: No module named setuptools_rust
sonsangwoo@sonui-Macmini python % pip3 install setuptools_rust
Requirement already satisfied: setuptools_rust in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (1.3.0)
Requirement already satisfied: semantic-version<3,>=2.8.2 in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from setuptools_rust) (2.9.0)
Requirement already satisfied: setuptools>=58.0 in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from setuptools_rust) (62.3.2)
Requirement already satisfied: typing-extensions>=3.7.4.3 in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from setuptools_rust) (4.2.0)
sonsangwoo@sonui-Macmini python % pip install tokenizers                                        
Collecting tokenizers
  Using cached tokenizers-0.12.1.tar.gz (220 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: tokenizers
  Building wheel for tokenizers (pyproject.toml) ... done
  Created wheel for tokenizers: filename=tokenizers-0.12.1-cp310-cp310-macosx_12_0_universal2.whl size=3569749 sha256=46ea5f4849bff346fcec7ecb71e08676f4a1b77e449265a978d3774b175b55e0
  Stored in directory: /Users/sonsangwoo/Library/Caches/pip/wheels/bd/22/bc/fa8337ce1ccf384c8fc4c1dbfa9cb1687934c0f24719082d49
Successfully built tokenizers
Installing collected packages: tokenizers
Successfully installed tokenizers-0.12.1

 

transformers