Wednesday, November 12, 2008


Can my computer pass my exams?

I am taking a Chinese module this semester and after a long while actively continue my long struggle to master this amazingly beautiful and also amazingly difficult language. My last post that was in Chinese was a homework for that course.

So today there was the final exam for this semester. The exam consisted of the following parts that each contributed 20% of the total score:

1. Make sentences: given a Chinese word make a sentence with this word
2. Reading comprehension: read a short story and answer the questions
3. Spot the error: find the error in a sentence and correct it
4. Word jumble: re-arrange the words to form a correct sentence
5. Fill the blanks: Complete a sentence fragment using a given word

After finishing the exam, I started wondering, if I could actually build a computer program to solve the exam that I had just completed. And actually, I believe that most of the questions should be relatively easy to solve for a computer.

1. For making sentences, you could simply go and use a concordancer to collect a list of sentences that use the given word from a corpus or from the web. This seems a bit like cheating, as it does not involve any 'creative' aspect nor does it require a deeper semantic understanding of the word that is asked for. On the other hand, it seems to be okay for a student to memorize example sentences that he read before the test and put them in whenever it seems appropriate for him/her.

2. Reading comprehension is the the task that is probably the most difficult so far, because it requires two extremely difficult NLP tasks: natural language understanding and generation.
There is a whole bunch of work on these tasks, but the task is definitely hard. I really wonder if any NLP langauge understanding system could have done a job on the paper today.

3. Spot the error is a task that is already implemented in some commercial applications like Microsoft Word. So far most systems can only recognize a limited number of errors, for example determinor choice, but they can probably get 50%-60% of the questions in the exam correct.

4. To find the correct order of words is well-studied problem in machine translation (MT). Most MT systems translate the words and then try to solve the word re-arrangement problem in a consecutive step. One important model for this step is a n-gram language model, which basically computes the probability of a sentence by decomposing it into smaller sequences of length n where n is typically 2 or 3. I believe that a simple 3-gram model trained on a large enough Chinese corpus could have done a pretty good job on my exam today.

5. This step is somewhere between the first two tasks. Not sure, how well this works, but with enogh data it should probably be possible to solve this step by using a concordancer and a langauge model as well. Maybe it won't give the most meaningful sentences but probably good enough to come up with a well formed sentence.

This all, seems to be more a toy application, but it is an accepted standard to test the language competence of primary school students and second language learners by test questions like the above. If a computer could solve them just as well, it would show a certain langauge competence of the machine.


Comments: Post a Comment

<< Home

This page is powered by Blogger. Isn't yours?