MeCab-ipadic-NEologd をインストールしてみた
MeCab という形態素解析ツールがある。デフォルトの辞書は mecab-ipadic というモノで、MeCab 本体とともにインストールすると思うが、コレの改善版システム辞書である MeCab-ipadic-NEologd を試してみた。
検証環境は MacOS。brew install mecab mecab-ipadic で基本的な MeCab 環境はインストール済とする。
# どこでもいいがひとまず以下を作業ディレクトリとする
$ cd /usr/local/src/
# ソースを取得する
$ git clone --depth 1 https://github.com/neologd/mecab-ipadic-neologd.git
$ cd mecab-ipadic-neologd/
# 一括インストールする
$ ./bin/install-mecab-ipadic-neologd -n -a
[install-mecab-ipadic-NEologd] : Start..
[install-mecab-ipadic-NEologd] : Check the existance of libraries
[install-mecab-ipadic-NEologd] : find => ok
[install-mecab-ipadic-NEologd] : sort => ok
[install-mecab-ipadic-NEologd] : head => ok
[install-mecab-ipadic-NEologd] : cut => ok
[install-mecab-ipadic-NEologd] : egrep => ok
[install-mecab-ipadic-NEologd] : mecab => ok
[install-mecab-ipadic-NEologd] : mecab-config => ok
[install-mecab-ipadic-NEologd] : make => ok
[install-mecab-ipadic-NEologd] : curl => ok
[install-mecab-ipadic-NEologd] : sed => ok
[install-mecab-ipadic-NEologd] : cat => ok
[install-mecab-ipadic-NEologd] : diff => ok
[install-mecab-ipadic-NEologd] : tar => ok
[install-mecab-ipadic-NEologd] : unxz => ok
[install-mecab-ipadic-NEologd] : xargs => ok
[install-mecab-ipadic-NEologd] : grep => ok
[install-mecab-ipadic-NEologd] : iconv => ok
[install-mecab-ipadic-NEologd] : patch => ok
[install-mecab-ipadic-NEologd] : which => ok
[install-mecab-ipadic-NEologd] : file => ok
[install-mecab-ipadic-NEologd] : openssl => ok
[install-mecab-ipadic-NEologd] : awk => ok
[install-mecab-ipadic-NEologd] : mecab-ipadic-NEologd is already up-to-date
[install-mecab-ipadic-NEologd] : mecab-ipadic-NEologd will be install to /usr/local/lib/mecab/dic/mecab-ipadic-neologd
[install-mecab-ipadic-NEologd] : Make mecab-ipadic-NEologd
[make-mecab-ipadic-NEologd] : Start..
[make-mecab-ipadic-NEologd] : Check local seed directory
[make-mecab-ipadic-NEologd] : Check local seed file
[make-mecab-ipadic-NEologd] : Check local build directory
[make-mecab-ipadic-NEologd] : create /usr/local/src/mecab-ipadic-neologd/libexec/../build
[make-mecab-ipadic-NEologd] : Download original mecab-ipadic file
[make-mecab-ipadic-NEologd] : Try to access to https://ja.osdn.net
[make-mecab-ipadic-NEologd] : Try to download from https://ja.osdn.net/frs/g_redir.php?m=kent&f=mecab%2Fmecab-ipadic%2F2.7.0-20070801%2Fmecab-ipadic-2.7.0-20070801.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 11.6M 100 11.6M 0 0 3209k 0 0:00:03 0:00:03 --:--:-- 3396k
Hash value of /usr/local/src/mecab-ipadic-neologd/libexec/../build/mecab-ipadic-2.7.0-20070801.tar.gz matched
[make-mecab-ipadic-NEologd] : Decompress original mecab-ipadic file
x mecab-ipadic-2.7.0-20070801/
x mecab-ipadic-2.7.0-20070801/README
x mecab-ipadic-2.7.0-20070801/AUTHORS
x mecab-ipadic-2.7.0-20070801/COPYING
x mecab-ipadic-2.7.0-20070801/ChangeLog
x mecab-ipadic-2.7.0-20070801/INSTALL
x mecab-ipadic-2.7.0-20070801/Makefile.am
x mecab-ipadic-2.7.0-20070801/Makefile.in
x mecab-ipadic-2.7.0-20070801/NEWS
x mecab-ipadic-2.7.0-20070801/aclocal.m4
x mecab-ipadic-2.7.0-20070801/config.guess
x mecab-ipadic-2.7.0-20070801/config.sub
x mecab-ipadic-2.7.0-20070801/configure
x mecab-ipadic-2.7.0-20070801/configure.in
x mecab-ipadic-2.7.0-20070801/install-sh
x mecab-ipadic-2.7.0-20070801/missing
x mecab-ipadic-2.7.0-20070801/mkinstalldirs
x mecab-ipadic-2.7.0-20070801/Adj.csv
x mecab-ipadic-2.7.0-20070801/Adnominal.csv
x mecab-ipadic-2.7.0-20070801/Adverb.csv
x mecab-ipadic-2.7.0-20070801/Auxil.csv
x mecab-ipadic-2.7.0-20070801/Conjunction.csv
x mecab-ipadic-2.7.0-20070801/Filler.csv
x mecab-ipadic-2.7.0-20070801/Interjection.csv
x mecab-ipadic-2.7.0-20070801/Noun.adjv.csv
x mecab-ipadic-2.7.0-20070801/Noun.adverbal.csv
x mecab-ipadic-2.7.0-20070801/Noun.csv
x mecab-ipadic-2.7.0-20070801/Noun.demonst.csv
x mecab-ipadic-2.7.0-20070801/Noun.nai.csv
x mecab-ipadic-2.7.0-20070801/Noun.name.csv
x mecab-ipadic-2.7.0-20070801/Noun.number.csv
x mecab-ipadic-2.7.0-20070801/Noun.org.csv
x mecab-ipadic-2.7.0-20070801/Noun.others.csv
x mecab-ipadic-2.7.0-20070801/Noun.place.csv
x mecab-ipadic-2.7.0-20070801/Noun.proper.csv
x mecab-ipadic-2.7.0-20070801/Noun.verbal.csv
x mecab-ipadic-2.7.0-20070801/Others.csv
x mecab-ipadic-2.7.0-20070801/Postp-col.csv
x mecab-ipadic-2.7.0-20070801/Postp.csv
x mecab-ipadic-2.7.0-20070801/Prefix.csv
x mecab-ipadic-2.7.0-20070801/Suffix.csv
x mecab-ipadic-2.7.0-20070801/Symbol.csv
x mecab-ipadic-2.7.0-20070801/Verb.csv
x mecab-ipadic-2.7.0-20070801/char.def
x mecab-ipadic-2.7.0-20070801/feature.def
x mecab-ipadic-2.7.0-20070801/left-id.def
x mecab-ipadic-2.7.0-20070801/matrix.def
x mecab-ipadic-2.7.0-20070801/pos-id.def
x mecab-ipadic-2.7.0-20070801/rewrite.def
x mecab-ipadic-2.7.0-20070801/right-id.def
x mecab-ipadic-2.7.0-20070801/unk.def
x mecab-ipadic-2.7.0-20070801/dicrc
x mecab-ipadic-2.7.0-20070801/RESULT
[make-mecab-ipadic-NEologd] : Configure custom system dictionary on /usr/local/src/mecab-ipadic-neologd/libexec/../build/mecab-ipadic-2.7.0-20070801-neologd-20200130
checking for a BSD-compatible install... /usr/local/bin/ginstall -c
checking whether build environment is sane... yes
checking whether make sets $(MAKE)... yes
checking for working aclocal-1.4... missing
checking for working autoconf... found
checking for working automake-1.4... missing
checking for working autoheader... found
checking for working makeinfo... found
checking for a BSD-compatible install... /usr/local/bin/ginstall -c
checking for mecab-config... /usr/local/bin/mecab-config
configure: creating ./config.status
config.status: creating Makefile
[make-mecab-ipadic-NEologd] : Encode the character encoding of system dictionary resources from EUC_JP to UTF-8
./../../libexec/iconv_euc_to_utf8.sh ./Noun.place.csv
./../../libexec/iconv_euc_to_utf8.sh ./Auxil.csv
./../../libexec/iconv_euc_to_utf8.sh ./Noun.verbal.csv
./../../libexec/iconv_euc_to_utf8.sh ./Symbol.csv
./../../libexec/iconv_euc_to_utf8.sh ./Noun.org.csv
./../../libexec/iconv_euc_to_utf8.sh ./Noun.csv
./../../libexec/iconv_euc_to_utf8.sh ./Postp.csv
./../../libexec/iconv_euc_to_utf8.sh ./Adj.csv
./../../libexec/iconv_euc_to_utf8.sh ./Filler.csv
./../../libexec/iconv_euc_to_utf8.sh ./Noun.proper.csv
./../../libexec/iconv_euc_to_utf8.sh ./Noun.number.csv
./../../libexec/iconv_euc_to_utf8.sh ./Suffix.csv
./../../libexec/iconv_euc_to_utf8.sh ./Noun.others.csv
./../../libexec/iconv_euc_to_utf8.sh ./Interjection.csv
./../../libexec/iconv_euc_to_utf8.sh ./Noun.adjv.csv
./../../libexec/iconv_euc_to_utf8.sh ./Verb.csv
./../../libexec/iconv_euc_to_utf8.sh ./Others.csv
./../../libexec/iconv_euc_to_utf8.sh ./Adnominal.csv
./../../libexec/iconv_euc_to_utf8.sh ./Prefix.csv
./../../libexec/iconv_euc_to_utf8.sh ./Noun.demonst.csv
./../../libexec/iconv_euc_to_utf8.sh ./Adverb.csv
./../../libexec/iconv_euc_to_utf8.sh ./Noun.name.csv
./../../libexec/iconv_euc_to_utf8.sh ./Postp-col.csv
./../../libexec/iconv_euc_to_utf8.sh ./Conjunction.csv
./../../libexec/iconv_euc_to_utf8.sh ./Noun.nai.csv
./../../libexec/iconv_euc_to_utf8.sh ./Noun.adverbal.csv
rm ./Noun.place.csv
rm ./Auxil.csv
rm ./Noun.verbal.csv
rm ./Symbol.csv
rm ./Noun.org.csv
rm ./Noun.csv
rm ./Postp.csv
rm ./Adj.csv
rm ./Filler.csv
rm ./Noun.proper.csv
rm ./Noun.number.csv
rm ./Suffix.csv
rm ./Noun.others.csv
rm ./Interjection.csv
rm ./Noun.adjv.csv
rm ./Verb.csv
rm ./Others.csv
rm ./Adnominal.csv
rm ./Prefix.csv
rm ./Noun.demonst.csv
rm ./Adverb.csv
rm ./Noun.name.csv
rm ./Postp-col.csv
rm ./Conjunction.csv
rm ./Noun.nai.csv
rm ./Noun.adverbal.csv
./../../libexec/iconv_euc_to_utf8.sh ./rewrite.def
./../../libexec/iconv_euc_to_utf8.sh ./matrix.def
./../../libexec/iconv_euc_to_utf8.sh ./left-id.def
./../../libexec/iconv_euc_to_utf8.sh ./pos-id.def
./../../libexec/iconv_euc_to_utf8.sh ./unk.def
./../../libexec/iconv_euc_to_utf8.sh ./feature.def
./../../libexec/iconv_euc_to_utf8.sh ./right-id.def
./../../libexec/iconv_euc_to_utf8.sh ./char.def
rm ./rewrite.def
rm ./matrix.def
rm ./left-id.def
rm ./pos-id.def
rm ./unk.def
rm ./feature.def
rm ./right-id.def
rm ./char.def
mv ./Noun.others.csv.utf8 ./Noun.others.csv
mv ./Noun.number.csv.utf8 ./Noun.number.csv
mv ./Filler.csv.utf8 ./Filler.csv
mv ./Others.csv.utf8 ./Others.csv
mv ./unk.def.utf8 ./unk.def
mv ./Postp-col.csv.utf8 ./Postp-col.csv
mv ./Adnominal.csv.utf8 ./Adnominal.csv
mv ./Noun.verbal.csv.utf8 ./Noun.verbal.csv
mv ./matrix.def.utf8 ./matrix.def
mv ./Noun.csv.utf8 ./Noun.csv
mv ./Noun.demonst.csv.utf8 ./Noun.demonst.csv
mv ./char.def.utf8 ./char.def
mv ./Symbol.csv.utf8 ./Symbol.csv
mv ./Auxil.csv.utf8 ./Auxil.csv
mv ./Noun.name.csv.utf8 ./Noun.name.csv
mv ./feature.def.utf8 ./feature.def
mv ./Suffix.csv.utf8 ./Suffix.csv
mv ./Adverb.csv.utf8 ./Adverb.csv
mv ./Conjunction.csv.utf8 ./Conjunction.csv
mv ./pos-id.def.utf8 ./pos-id.def
mv ./Postp.csv.utf8 ./Postp.csv
mv ./right-id.def.utf8 ./right-id.def
mv ./Noun.nai.csv.utf8 ./Noun.nai.csv
mv ./Interjection.csv.utf8 ./Interjection.csv
mv ./Prefix.csv.utf8 ./Prefix.csv
mv ./Noun.place.csv.utf8 ./Noun.place.csv
mv ./Noun.adjv.csv.utf8 ./Noun.adjv.csv
mv ./rewrite.def.utf8 ./rewrite.def
mv ./Verb.csv.utf8 ./Verb.csv
mv ./left-id.def.utf8 ./left-id.def
mv ./Noun.proper.csv.utf8 ./Noun.proper.csv
mv ./Adj.csv.utf8 ./Adj.csv
mv ./Noun.adverbal.csv.utf8 ./Noun.adverbal.csv
mv ./Noun.org.csv.utf8 ./Noun.org.csv
[make-mecab-ipadic-NEologd] : Fix yomigana field of IPA dictionary
patching file Noun.csv
patching file Noun.place.csv
patching file Verb.csv
patching file Noun.verbal.csv
patching file Noun.name.csv
patching file Noun.adverbal.csv
patching file Noun.csv
patching file Noun.name.csv
patching file Noun.org.csv
patching file Noun.others.csv
patching file Noun.place.csv
patching file Noun.proper.csv
patching file Noun.verbal.csv
patching file Prefix.csv
patching file Suffix.csv
patching file Noun.proper.csv
patching file Noun.csv
patching file Noun.name.csv
patching file Noun.org.csv
patching file Noun.place.csv
patching file Noun.proper.csv
patching file Noun.verbal.csv
patching file Noun.name.csv
patching file Noun.org.csv
patching file Noun.place.csv
patching file Noun.proper.csv
patching file Suffix.csv
patching file Noun.demonst.csv
patching file Noun.csv
patching file Noun.name.csv
[make-mecab-ipadic-NEologd] : Copy user dictionary resource
[make-mecab-ipadic-NEologd] : Install adverb entries using /usr/local/src/mecab-ipadic-neologd/libexec/../seed/neologd-adverb-dict-seed.20150623.csv.xz
[make-mecab-ipadic-NEologd] : Install interjection entries using /usr/local/src/mecab-ipadic-neologd/libexec/../seed/neologd-interjection-dict-seed.20170216.csv.xz
[make-mecab-ipadic-NEologd] : Install noun orthographic variant entries using /usr/local/src/mecab-ipadic-neologd/libexec/../seed/neologd-common-noun-ortho-variant-dict-seed.20170228.csv.xz
[make-mecab-ipadic-NEologd] : Install noun orthographic variant entries using /usr/local/src/mecab-ipadic-neologd/libexec/../seed/neologd-proper-noun-ortho-variant-dict-seed.20161110.csv.xz
[make-mecab-ipadic-NEologd] : Install entries of orthographic variant of a noun used as verb form using /usr/local/src/mecab-ipadic-neologd/libexec/../seed/neologd-noun-sahen-conn-ortho-variant-dict-seed.20160323.csv.xz
[make-mecab-ipadic-NEologd] : Install frequent adjective orthographic variant entries using /usr/local/src/mecab-ipadic-neologd/libexec/../seed/neologd-adjective-std-dict-seed.20151126.csv.xz
[make-mecab-ipadic-NEologd] : Install infrequent adjective orthographic variant entries using /usr/local/src/mecab-ipadic-neologd/libexec/../seed/neologd-adjective-exp-dict-seed.20151126.csv.xz
[make-mecab-ipadic-NEologd] : Install adjective verb orthographic variant entries using /usr/local/src/mecab-ipadic-neologd/libexec/../seed/neologd-adjective-verb-dict-seed.20160324.csv.xz
[make-mecab-ipadic-NEologd] : Install infrequent datetime representation entries using /usr/local/src/mecab-ipadic-neologd/libexec/../seed/neologd-date-time-infreq-dict-seed.20190415.csv.xz
[make-mecab-ipadic-NEologd] : Install infrequent quantity representation entries using /usr/local/src/mecab-ipadic-neologd/libexec/../seed/neologd-quantity-infreq-dict-seed.20190415.csv.xz
[make-mecab-ipadic-NEologd] : Install entries of ill formed words using /usr/local/src/mecab-ipadic-neologd/libexec/../seed/neologd-ill-formed-words-dict-seed.20170127.csv.xz
[make-mecab-ipadic-NEologd] : Re-Index system dictionary
reading ./unk.def ... 40
emitting double-array: 100% |###########################################|
./model.def is not found. skipped.
reading ./neologd-adjective-verb-dict-seed.20160324.csv ... 20268
reading ./Noun.place.csv ... 73194
reading ./Auxil.csv ... 199
reading ./Noun.verbal.csv ... 12150
reading ./Symbol.csv ... 208
reading ./Noun.org.csv ... 17149
reading ./Noun.csv ... 60734
reading ./Postp.csv ... 146
reading ./neologd-ill-formed-words-dict-seed.20170127.csv ... 60616
reading ./Adj.csv ... 27210
reading ./Filler.csv ... 19
reading ./Noun.proper.csv ... 27493
reading ./Noun.number.csv ... 42
reading ./Suffix.csv ... 1448
reading ./Noun.others.csv ... 153
reading ./Interjection.csv ... 252
reading ./mecab-user-dict-seed.20200130.csv ... 3179078
reading ./Noun.adjv.csv ... 3328
reading ./Verb.csv ... 130750
reading ./neologd-date-time-infreq-dict-seed.20190415.csv ... 16866
reading ./neologd-proper-noun-ortho-variant-dict-seed.20161110.csv ... 138379
reading ./neologd-adjective-exp-dict-seed.20151126.csv ... 1051146
reading ./Others.csv ... 2
reading ./Adnominal.csv ... 135
reading ./neologd-common-noun-ortho-variant-dict-seed.20170228.csv ... 152869
reading ./neologd-quantity-infreq-dict-seed.20190415.csv ... 229216
reading ./neologd-noun-sahen-conn-ortho-variant-dict-seed.20160323.csv ... 26058
reading ./neologd-adjective-std-dict-seed.20151126.csv ... 507812
reading ./Prefix.csv ... 224
reading ./Noun.demonst.csv ... 120
reading ./Adverb.csv ... 3032
reading ./neologd-adverb-dict-seed.20150623.csv ... 139792
reading ./neologd-interjection-dict-seed.20170216.csv ... 4701
reading ./Noun.name.csv ... 34215
reading ./Postp-col.csv ... 91
reading ./Conjunction.csv ... 171
reading ./Noun.nai.csv ... 42
reading ./Noun.adverbal.csv ... 808
emitting double-array: 100% |###########################################|
reading ./matrix.def ... 1316x1316
emitting matrix : 100% |###########################################|
done!
[make-mecab-ipadic-NEologd] : Make custom system dictionary on /usr/local/src/mecab-ipadic-neologd/libexec/../build/mecab-ipadic-2.7.0-20070801-neologd-20200130
make: Nothing to be done for `all'.
[make-mecab-ipadic-NEologd] : Finish..
[install-mecab-ipadic-NEologd] : Get results of tokenize test
[test-mecab-ipadic-NEologd] : Start..
[test-mecab-ipadic-NEologd] : Replace timestamp from 'git clone' date to 'git commit' date
[test-mecab-ipadic-NEologd] : Get buzz phrases
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1479 100 1479 0 0 7085 0 --:--:-- --:--:-- --:--:-- 7110
[test-mecab-ipadic-NEologd] : Get difference between default system dictionary and mecab-ipadic-NEologd
[test-mecab-ipadic-NEologd] : Tokenize phrase using default system dictionary
[test-mecab-ipadic-NEologd] : Tokenize phrase using mecab-ipadic-NEologd
[test-mecab-ipadic-NEologd] : Get result of diff
[test-mecab-ipadic-NEologd] : Please check difference between default system dictionary and mecab-ipadic-NEologd
default system dictionary | mecab-ipadic-NEologd
さかな クン | さかなクン
虚 血 性 心不全 | 虚血性 心不全
天神 ビブレ | 天神ビブレ
徹子 の 部屋 | 徹子の部屋
新庄 剛志 | 新庄剛志
野村 監督 | 野村監督
ニューサウスウェールズ 州 | ニューサウスウェールズ州
バネ ブー | バネブー
宮 根 | 宮根
タナ トス | タナトス
野村 克也 | 野村克也
佐賀 記念 | 佐賀記念
野村 沙 知代 | 野村沙知代
[test-mecab-ipadic-NEologd] : Finish..
[install-mecab-ipadic-NEologd] : Please check the list of differences in the upper part.
[install-mecab-ipadic-NEologd] : Do you want to install mecab-ipadic-NEologd? Type yes or no.
yes
[install-mecab-ipadic-NEologd] : OK. Let's install mecab-ipadic-NEologd.
[install-mecab-ipadic-NEologd] : Start..
[install-mecab-ipadic-NEologd] : /usr/local/lib/mecab/dic is current user's directory
[install-mecab-ipadic-NEologd] : Make install to /usr/local/lib/mecab/dic/mecab-ipadic-neologd
make[1]: Nothing to be done for `install-exec-am'.
/bin/sh ./mkinstalldirs /usr/local/lib/mecab/dic/mecab-ipadic-neologd
mkdir /usr/local/lib/mecab/dic/mecab-ipadic-neologd
/usr/local/bin/ginstall -c -m 644 ./matrix.bin /usr/local/lib/mecab/dic/mecab-ipadic-neologd/matrix.bin
/usr/local/bin/ginstall -c -m 644 ./char.bin /usr/local/lib/mecab/dic/mecab-ipadic-neologd/char.bin
/usr/local/bin/ginstall -c -m 644 ./sys.dic /usr/local/lib/mecab/dic/mecab-ipadic-neologd/sys.dic
/usr/local/bin/ginstall -c -m 644 ./unk.dic /usr/local/lib/mecab/dic/mecab-ipadic-neologd/unk.dic
/usr/local/bin/ginstall -c -m 644 ./left-id.def /usr/local/lib/mecab/dic/mecab-ipadic-neologd/left-id.def
/usr/local/bin/ginstall -c -m 644 ./right-id.def /usr/local/lib/mecab/dic/mecab-ipadic-neologd/right-id.def
/usr/local/bin/ginstall -c -m 644 ./rewrite.def /usr/local/lib/mecab/dic/mecab-ipadic-neologd/rewrite.def
/usr/local/bin/ginstall -c -m 644 ./pos-id.def /usr/local/lib/mecab/dic/mecab-ipadic-neologd/pos-id.def
/usr/local/bin/ginstall -c -m 644 ./dicrc /usr/local/lib/mecab/dic/mecab-ipadic-neologd/dicrc
[install-mecab-ipadic-NEologd] : Install completed.
[install-mecab-ipadic-NEologd] : When you use MeCab, you can set '/usr/local/lib/mecab/dic/mecab-ipadic-neologd' as a value of '-d' option of MeCab.
[install-mecab-ipadic-NEologd] : Usage of mecab-ipadic-NEologd is here.
Usage:
$ mecab -d /usr/local/lib/mecab/dic/mecab-ipadic-neologd ...
[install-mecab-ipadic-NEologd] : Finish..
[install-mecab-ipadic-NEologd] : Finish..
…とまぁこんな感じでインストールができた。
Python コード内で使うには
インストールしたシステム辞書を Python コード内で使うには、以下のようにする。
import MeCab
# 通常以下のようにしていたところを…
#mecab = MeCab.Tagger()
# 次のように、システム辞書のインストール先を示しながらインスタンス生成する
mecab = MeCab.Tagger('-d /usr/local/lib/mecab/dic/mecab-ipadic-neologd')
parsed_nodes = mecab.parseToNode('こんにちは、私は Neo です。')
# 以下略…
-d オプションで示しているパスはインストール中に表示されているので確認しておこう。
以上。